System and method for computer based testing using cache and cacheable objects to expand functionality of a test driver application

ABSTRACT

A system and method for computer-based testing includes a test driver that controls delivery of a computer-based test to one or more test candidates and that controls caching of test components during delivery of the test. The system includes various monitoring components, including monitoring of candidate progress, candidate performance, network bandwidth, network latency and server response, during delivery of the test and adjusting the source of the test components or the volume of the test components being cached for delivery of the test. Based upon this monitoring of the system, for example, if network communication failure is detected, the test candidate is able to continue computer-based testing while connectivity is being reestablished.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to and claims priority from U.S. ProvisionalPatent Application No. 60/479,952, filed Jun. 20, 2003, the disclosureof which is incorporated herein by reference in its entirety, and isfurther related to U.S. Patent Publication No. 20030203342, published onOct. 30, 2003 and entitled “METHOD AND SYSTEM FOR COMPUTER BASED TESTINGUSING CUSTOMIZABLE TEMPLATES”, U.S. Patent Publication No. 20030196170,published on Oct. 16, 2003 and entitled “METHOD AND SYSTEM FOR COMPUTERBASED TESTING USING A NON-DETERMINISTIC EXAM EXTENSIBLE LANGUAGE (XXL)PROTOCOL”, U.S. Patent Publication No. 20030182602, published on Sep.25, 2003 and entitled “METHOD AND SYSTEM FOR COMPUTER BASED TESTINGUSING PLUGINS TO EXPAND FUNCTIONALITY OF A TEST DRIVER”, and U.S. PatentPublication No. 20030138765, published on Jul. 24, 2003 and entitled“METHOD AND SYSTEM FOR COMPUTER BASED TESTING USING AN AMALGAMATEDRESOURCE FILE”, and U.S. Patent Publication No. 20030129573, publishedon Jul. 10, 2003 and entitled “EXTENSIBLE EXAM LANGUAGE (XXL) PROTOCOLFOR COMPUTER BASED TESTING”, all of which were filed concurrently andall of which are incorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is related to systems and methods used tofacilitate computer based testing such as those that utilize networksystems. More particularly, the present invention uses cacheable objectsto expand functionality of a test driver application. And, even moreparticularly, such cacheable objects include cacheable data objects andcacheable program and application objects that may be used by a testdriver application in facilitating test taking and administration.

2. Description of the Related Art

For many years, standardized testing has been a common method ofassessing examinees as regards educational placement, skill evaluation,etc. Due to the prevalence and mass distribution of standardized tests,computer-based testing has emerged as a superior method for providingstandardized tests, guaranteeing accurate scoring, and ensuring promptreturn of test results to examinees.

Tests are developed based on the requirements and particulars of testdevelopers. Typically, test developers employ psychometricians orstatisticians and psychologists to determine the specific requirementsspecific to human assessment. These experts often have their own, uniqueideas regarding how a test should be presented and regarding thenecessary contents of that test, including the visual format of the testas well as the data content of the test. Therefore, a particularcomputer-based test has to be customized to fulfill the client'srequirements.

FIG. 1 illustrates a prior art process for computerized testcustomization, denoted generally by reference numeral 10. First, aclient details the desired test requirements and specifications, step12. The computerized test publisher then creates the tools that allowthe test publisher to author the items, presentations, etc., required tofulfill the requirements, step 14. The test publisher then writes anitem viewer, which allows the test publisher to preview what is beingauthored, step 16.

An item presenter is then written to present the new item, for example,to the test driver, step 18. Presenting the new item to the test driverrequires a modification of the test driver's executable code. The testdriver must be modified so that it is aware of the new item and cancommunicate with the new item presenter, step 20. The test packager mustthen also be modified, step 22. The test packager, which may also be acompiler, takes what the test publisher has created and writes theresult as new object codes for the new syntax. Subsequently, the scoringengine must also be modified to be able to score the new item type, step24. Finally, the results processor must be modified to be able to acceptthe new results from the new item, step 26. This process requires noless than seven software creations or modifications to existingsoftware.

U.S. Pat. No. 5,827,070 (Kershaw et al.) and U.S. Pat. No. 5,565,316(Kershaw et al.) are incorporated herein by reference. The '070 and '316patents, which have similar specifications, disclose a computer-basedtesting system comprising a test development system and a test deliverysystem. The test development system comprises a test document creationsystem for specifying the test contents, an item preparation system forcomputerizing each of the items in the test, a test preparation systemfor preparing a computerized test, and a test packaging system forcombining all of the items and test components into a computerized testpackage. The computerized test package is then delivered to authorizedexaminees on a workstation by the test delivery system.

FIGS. 2A and 2B illustrate the test preparation process as disclosed inthe '070 and '316 patents. Test developers assemble the test as shown at32. As shown at 36, item selection is preferably automated (AIS) usingthe test development/document creation (“TD/DC”) system or an equivalenttest document creation system. Using “TD/DC”, test developers enter thetest specifications into the “TD/DC” system. Based on thesespecifications, “TD/DC” searches its central database for items, whichsatisfy the test specification, e.g., 50 math questions, 25 of which arealgebra problems and 25, which are geometry problems. Then, the testdevelopers review the items selected by “TD/DC” for sensitivity andoverlap constraints described in the background section. If the testdeveloper decides that the sensitivity or overlap constraints are notsatisfied by the current selection of items, certain items may bedesignated to be replaced by another item from the database. Inaddition, test developers provide a test description specifying thedirections, messages, timing of sections, number of sections of thetest, etc. as shown at 42. If a computer adaptive test (CAT) is to berun, test developers may run a computer adaptive test simulation at 34,which are known to skilled test developers. Using the Test PreparationTool (TPT) and TOOLBOOK 46, the test preparation system (“TPS”) preparesthe test level components as shown at 50. TOOLBOOK is commerciallyavailable from Asymetrix Corporation. The test level components includescripts 66, item table block sets 56, general information screens 58,direction screens 60, message screens 62, and tutorial units 64. Each ofthe test components will be described in detail below. As the componentsare prepared, the TPT stores them in a TPS network directory 52. Then,the components are entered into the TPS Production database 54. Thecomponents stored in the TPS Production database 54 will be retrievedduring test packaging.

U.S. Pat. No. 5,513,994 (Kershaw et al.), which is incorporated hereinby reference, discloses a centralized administrative system and methodof administering standardized test to a plurality of examinees. Theadministrative system is implemented on a central administrationworkstation and at least one test workstation located in different roomsat a test center. The administrative system software, which providessubstantially administrative functions, is executed from the centraladministration workstation. The administrative system software, whichprovides function carried out in connection with a test session, isexecuted from the testing workstations.

However, computer-based testing has expanded from standalonedistribution administered at a local test center to wide area networkdistribution administered via clustered servers at multiple locations.Thus, a distributed computer-based testing system requires scalabilityto support continuous exam administration and a high volume ofconcurrent test candidates who may be located at many remote locations.

Additionally, computer-based tests have evolved from mere display ofsimple text-based content to include streaming of audio and videocontent. Thus, a distributed computer-based testing system demandssufficient system resources and storage capacity as well as efficientdata communication management to serve bandwidth-intensive multimediacontent in a consistent manner.

Moreover, computer-based test models have advanced to include adaptiveand simulation test models. Thus, a distributed computer-based testingsystem must support a variety of complex test models.

Further, a distributed computer-based testing system must facilitate afair testing environment within a dynamic networked environment to testcandidates who may have varying workstation capabilities or networkconnectivity. A number of factors affect the creation and maintenance ofa fair testing environment, including bandwidth mismatches and networklatency between a test candidate workstation and a test distributionserver as well as between a test distribution server and a test sourceserver, the available system resources of the test source server, thetest distribution servers and the test candidate workstations, and testcomponent characteristics (e.g., whether the object is text, audio orvideo). Thus, it is necessary to monitor candidate progress, candidateperformance, network bandwidth, network latency, and server response,among other testing environment variables, during computer-based testingand cache test components in response to changes in the testingenvironment in order to ensure timely and consistent delivery of thecomputer-based test. In other words, a distributed computer-basedtesting system must be adjustable to emulate a suitable testingenvironment on test candidate workstations concurrently executing thesame computer-based test.

SUMMARY OF THE INVENTION

The present invention discloses a computer-based testing system thatcontrols delivery of a computer-based test to a high volume ofconcurrent test candidates and that adapts delivery of thecomputer-based test in response to changes in the testing environment.

It is one feature and advantage of the present invention to delivercomputer-based tests to a high volume of concurrent test candidateslocated at multiple locations.

It is another feature and advantage of the present invention to securelyadminister computer-based tests among concurrent test candidates locatedat multiple locations.

It is another feature and advantage of the present invention to monitorcandidate progress during computer-based testing for ensuring that testcomponents are timely available for delivery to a test candidate duringtesting.

It is another feature and advantage of the present invention to monitorcandidate performance during computer-based testing for ensuring thatsuitable test components are available for delivery to a test candidateduring testing, for example, to support Computer Adaptive Testing (CAT).

It is another feature and advantage of the present invention to monitornetwork bandwidth during computer-based testing for adapting delivery ofa computer-based test to a test candidate in accordance with the networkbandwidth.

It is another feature and advantage of the present invention to monitornetwork latency during computer-based testing for adapting delivery of acomputer-based test to a test candidate in accordance with the networklatency.

It is another feature and advantage of the present invention to monitorserver response during computer-based testing for adapting delivery of acomputer-based test to a test candidate in accordance with the serverresponse.

It is another feature and advantage of the present invention to enable atest candidate to launch a computer-based test on the candidate'sworkstation prior to all the test components having been delivered tothe candidate workstation.

It is another feature and advantage of the present invention to enable atest candidate to continue computer-based testing when networkconnectivity fails during computer-based testing.

These and other features and advantages of the present invention areachieved in systems and methods for computer-based testing including atest driver application that controls delivery of a computer-based testto a test candidate. Particularly, the system includes cacheableobjects, including cacheable data objects (test items) and cacheableprogram and application objects (plugins), collectively, testcomponents, to expand the functionality of the test driver application,enabling the test driver application to control caching of testcomponents in response to changes in the testing environment duringdelivery of a computer-based test to a candidate workstation. Thesystems and methods include monitoring candidate progress, candidateperformance, network bandwidth, network latency and server responseduring delivery of the computer-based test and adjusting either thesource of test components or the volume of test components being cachedfor delivery of the test. Based upon such monitoring, for example, ifnetwork communication failure is detected, the test candidate is able tocontinue computer-based testing while connectivity is beingreestablished in the background.

There has thus been outlined, rather broadly, the more importantfeatures of the invention and the preferred embodiments in order thatthe detailed description thereof that follows may be better understood,and in order that the present contribution to the art may be betterappreciated. There are, of course, additional features of the inventionthat will be described hereinafter and which will form the subjectmatter of the claims appended hereto.

In this respect, before explaining the preferred embodiments of theinvention in detail, it is to be understood that the invention is notlimited in its application to the details of construction and to thearrangements of the components set forth in the following description orillustrated in the drawings. The invention is capable of otherembodiments and of being practiced and carried out in various ways.Also, it is to be understood that the phraseology and terminologyemployed herein are for the purpose of description and should not beregarded as limiting.

As such, those skilled in the art will appreciate that the conception,upon which this disclosure is based, may readily be utilized as a basisfor the designing of other structures, methods and systems for carryingout the several purposes of the present invention. It is important,therefore, that the claims be regarded as including such equivalentconstructions insofar as they do not depart from the spirit and scope ofthe present invention.

Further, the purpose of the foregoing abstract is to enable the U.S.Patent and Trademark Office and the public generally, and especially thescientists, engineers and practitioners in the art who are not familiarwith patent or legal terms or phraseology, to determine quickly from acursory inspection the nature and essence of the technical disclosure ofthe application. The abstract is neither intended to define theinvention of the application, which is measured by the claims, nor is itintended to be limiting as to the scope of the invention in any way.

These, together with other objects of the invention, along with thevarious features of novelty, which characterize the invention, arepointed out with particularity in the claims annexed to and forming apart of this disclosure. For a better understanding of the invention,its operating advantages and the specific objects attained by its uses,reference should be had to the accompanying drawings and descriptivematter in which there is illustrated preferred embodiments of theinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram of a prior art method of computerized testcustomization;

FIGS. 2A and 2B are block diagrams of a prior art process for testproduction;

FIG. 3 is a schematic diagram of a computer-based testing system;

FIG. 4 is a block diagram illustrating different types of plugins thatare used with the computer-based testing system;

FIG. 5 illustrates various components that comprise an exam source file;

FIGS. 6A and 6B are a schematic illustrating the components, classes,and interfaces that comprise a test definition language compiler;

FIG. 7 is a schematic illustrating the components that comprise a testdriver and a test administration system;

FIGS. 8A and 8B are schematics illustrating the classes and interfacesthat comprise the test driver;

FIG. 9 illustrating the interfaces that comprise a structured storage;

FIGS. 10A and 10B are schematics illustrating the classes and interfacesthat comprise the structure storage and associated operations;

FIG. 11 is a block diagram of main storage branches of an exam resourcefile;

FIG. 12 is a block diagram illustrating an exams branch of the examresource file;

FIGS. 13A and 13B are block diagrams illustrating a forms branch of theexam resource file;

FIG. 14 is a block diagram illustrating an items branch of the examresource file;

FIG. 15 is a block diagram illustrating a categories branch of the examresource file;

FIG. 16 is a block diagram illustrating a templates branch of the examresource file;

FIG. 17 is a block diagram illustrating a sections branch of the examresource file;

FIG. 18 is a block diagram illustrating a groups branch of the examresource file;

FIGS. 19A, 19B, 19C, and 19D are block diagrams illustrating an eventssub-branch of the groups branch of the exam resource file;

FIG. 20 is a block diagram illustrating a plugins branch of the examresource file;

FIG. 21 is a block diagram illustrating a data branch of the examresource file;

FIG. 22 is a block diagram illustrating a formGroups branch of the examresource file;

FIG. 23 is a block diagram illustrating an attributes branch of the examresource file;

FIG. 24 is a block diagram illustrating a scripts branch of the examresource file;

FIG. 25 is a block diagram illustrating a message box branch of the examresource file;

FIGS. 26A, 26B, 26C, and 26D are block diagrams of an exam instancefile;

FIG. 27 is a flow diagram of a method of computerized testcustomization;

FIG. 28 is a flow chart of a method of test production and testdelivery;

FIG. 29 is a flow chart of a method for validation of test specificationand content;

FIG. 30 is a flow chart of a method for test delivery;

FIG. 31 is a flow chart of a method of restarting a test afterinterruption;

FIG. 32 is a diagram of a life cycle of a plugin;

FIG. 33 is a flow diagram of a process for compiling plugins;

FIGS. 34A, 34B, 34C, and 34D are flow diagrams of a process fordelivering plugins to an examinee during a computer-based test;

FIG. 35 is a block diagram illustrating an example of a networkenvironment for a computer-based testing system according to the presentinvention;

FIG. 36 is a block diagram illustrating a caching architecture for acomputer-based testing system according to the present invention;

FIGS. 37A and 37B are flow charts of methods for caching test componentsaccording to the present invention; and

FIGS. 38A, 38B, 38C, 38D and 38E are flow charts of methods formonitoring candidate performance, candidate progress, network latency,network bandwidth and server response according to the presentinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference now will be made in detail to the presently preferredembodiments of the invention. Such embodiments are provided by way ofexplanation of the invention, which is not intended to be limitedthereto. In fact, those of ordinary skill in the art may appreciate uponreading the present specification and viewing the present drawings thatvarious modifications and variations can be made.

For example, features illustrated or described as part of one embodimentcan be used on other embodiments to yield a still further embodiment.Additionally, certain features may be interchanged with similar devicesor features not mentioned yet which perform the same or similarfunctions. It is therefore intended that such modifications andvariations are included within the totality of the present invention.

The present invention discloses a system and method of computer-basedtesting using a test driver that is, for example, object-oriented and isarchitected to dynamically add functionality through, for example, theuse of an expansion module, and preferably through the use of plugins.The test driver preferably references component object model serversusing standard interfaces, and uses, for example, class names (that canbe an Active Document) defined in a custom test definition languageentitled extensible exam Language (“XXL”) based on extensible MarkupLanguage (“XML”) format to interact with existing applications whileoffering the flexibility of allowing development of new plugins. Thesenew plugins can be customized to a client's needs without changing thecore test driver. The specific format and protocol of XXL is alsodescribed in U.S. Patent Publication No. 20030129573, published Jul. 10,2003 and entitled “EXTENSIBLE EXAM LANGUAGE (XXL) PROTOCOL FOR COMPUTERBASED TESTING”, incorporated herein by reference.

The plugins advantageously enable the test driver to support, forexample, new item types, navigation algorithms, information displays,scoring algorithms, timing algorithms, test unit selection algorithms,results persistence reporting, printed score reporting, and/or helmtypes without change to the test driver's executable. Plugins also allowexpansion of the test driver's functionality without requiring the testdriver to be recompiled or re-linked, and without requiring the testpublisher to learn to program. Since plugins are written independentlyof the test driver, plugins can be written long after the test driver isbuilt. The client and the software developer can design and test theplugins and distribute the plugins to each test site. By using thismethod, large-scale regression testing of other examinations will notusually be necessary unless changes are made to the plugins that may beused by many examinations. The specific use of plugins is described inU.S. Patent Publication No. 20030182602, published Sep. 25, 2003 andentitled “METHOD AND SYSTEM FOR COMPUTER BASED TESTING USING PLUGINS TOEXPAND FUNCTIONALITY OF A TEST DRIVER”, incorporated herein byreference.

The test driver of the present invention controls delivery of acomputer-based test to a test candidate, including controlling cachingof test components during delivery of the test. In accordance tomonitoring of the testing environment, including monitoring candidateprogress, candidate performance, network bandwidth, network latency andserver response, during delivery of the test, the test driver adjustseither the source of test components or the volume of test componentsbeing cached for delivery of the test. Based on such monitoring, forexample, if network communication failure is detected, the testcandidate is able to continue computer-based testing while connectivityis being reestablished in the background. By using this system, auniform testing environment can be established and maintained duringcomputer-based testing in a distributed computer-based testingenvironment.

I. Overview of Computer-Based Test Delivery System

FIG. 3 shows an overview of the software architecture for thecomputer-based test delivery system of the present invention, denotedgenerally by reference numeral 100. Test driver 110 is responsible forcontrolling all aspects of the computer-based test. Test driver 110identifies candidates scheduled to take the computer-based test andidentifies and creates the appropriate test. Test driver 110 thenpresents all of the test components to candidates using a display device(not shown), such as a computer monitor, and enables candidates to enterresponses to test questions through the use of an input device (notshown), such as a keyboard, a mouse, etc. Test driver 110 also monitorsthe security of the test. For example, test driver 110 can preventaccess to the Internet and can validate candidates, although, thesefunctions are preferably performed by the test center administrationsystem. Test driver 110 also monitors the timing of the test, providingrelevant warnings to candidate regarding the elapsed time of the testand the time remaining for a particular section of the test or for theentire test. Test driver 110 is also responsible for scoring the test,once the test is completed or while the test is in progress, and forreporting the results of the test by physical printout using printer 182or in a file format using candidate exam results file 180. If the testis interrupted while in progress, for example, due to a power failure,test driver 110 restarts the test, preferably at the point at which thetest was interrupted, as will be described subsequently in more detail.Finally, if the test is left incomplete, test driver 110 cleans up theincomplete test. An incomplete test will have an exam instance file inthe candidate's directory but will not have created a results file. Aresults file is created even though generally the candidate will fail.The number of items delivered to the candidate is recorded in theresults file. Test driver 110 picks up where the event was interruptedand invisibly delivers the rest of the units of the test.

A test specification is authored by a test publisher according to thespecifications of the client and stored in exam source files 130. Examsource files 130 include data files 132, XXL files 134, multimedia files136, and hypertext markup language (“HTML”) files 138. XXL files 134include the test specification, which contains the client's requirementsfor the test, a bank of test items or questions, templates thatdetermine the physical appearance of the test, plugins, and anyadditional data necessary to implement the test. Additional data is alsostored in data files 132. For example an adaptive selection plugin mayneed a, b & c theta values. These values are stored in a binary filecreated by a statistical package.

HTML files 130 include, for example, any visual components of the test,such as the appearance of test items or questions, the appearance ofpresentations on the display device, the appearance of any clientspecified customizations, and/or the appearance of score reports. HTMLfiles 130 preferably also include script, for example, VBscript andJscript, or Java script. HTML files 130 are preferably authored usingMicrosoft's FrontPage 2000. FrontPage 2000 is preferably also used tomanage the source files in a hierarchy that is chosen by the testpublisher. Multimedia files 136 include, for example, any images (.jpg,.gif, etc.) and/or sound files (.mp3, .wav, .au, etc.) that are usedduring the test.

XXL compiler 140 retrieves XXL files 134 from exam source files 130using interface 190 and compiles the XXL test content stored in XXLfiles 134. XXL compiler 140 stores the compiled test files in examresource file 120. In another embodiment, exam source files 130 do notcontain XXL files 134 and contains, for example, only multi-media files.In this embodiment, XXL compiler 140 is merely a test packager thatwrites the data directly to exam resource file 120 without modificationor validation. The data appears in a stream under the “data” branch ofexam resource file 120. The name of the stream is specified by the testauthor.

In a preferred embodiment, XXL files 134 also include XXL language thatdefines plugins 150, in which case, plugins 150 assist XXL compiler 140in compiling XXL files 134. Test driver 110 preferably supports, forexample, nine different types of plugins 150, including, for example:display plugin 152; helm plugin 154; item plugin 156; timer plugin 158;selection plugin 160; navigation plugin 162; scoring plugin 164; resultsplugin 166; and report plugin 168. Plugins 150, which are also includedin XXL files 134, are the first XML files compiled into exam resourcefile 120.

Plugins 150 allow a test designer to customize the behavior of testdriver 110 and are divided into two types, for example: visible pluginsand invisible plugins, as shown in FIG. 4. The visible plugins, whichinclude display plugin 152, helm plugin 154, and item plugin 156, enablethe test driver to control what is presented visually to an candidate onthe display device. The invisible plugins, which include timer plugin158, selection plugin 160, navigation plugin 162, scoring plugin 164,results plugin 166, and report plugin 168, enable the test driver tocontrol more functional aspects of the test. Plugins 150 are used tovalidate data stored in exam source files 130 that is to be used by oneof plugins 150 during delivery of the test to the candidate, as isdescribed below in greater detail. Plugins 150 are, preferably,component object model (“COM”) objects, as described below. Plugins 150,may also utilize Java implementation. Plugins 150 are preferably writtenusing Microsoft Visual C++ or Visual Basic 6.0 or any fully COM enabledlanguage. Plugins 150 may be in or out-of-process, and, therefore, canexist as executable (“.EXE”) files or as dynamic link library (“.DLL”)files.

An application or component that uses objects provided by anothercomponent is called a client. Components are characterized by theirlocation relative to clients. An out-of process component is an .exefile that runs in its own process, with its own thread of execution.Communication between a client and an out-of-process component istherefore called cross-process or out-of-process communication.

An in-process component, such as a .dll or .ocx file, runs in the sameprocess as the client. It provides the fastest way of accessing objects,because property and method calls don't have to be marshaled acrossprocess boundaries. However, an in-process component must use theclient's thread of execution.

Exam resource file 120 receives the compiled test content from XXLcompiler 140 and plugins 150, if applicable, and stores the compiledtest content in an object-linking and embedding (“OLE”) structuredstorage format, called POLESS, which is described in greater detailbelow. Other storage formats may optionally be used. OLE allowsdifferent objects to write information into the same file, for example,embedding an Excel spreadsheet inside a Word document. OLE supports twotypes of structures, embedding and linking. In OLE embedding, the Worddocument of the example is a container application and the Excelspreadsheet is an embedded object. The container application contains acopy of the embedded object, and changes made to the embedded objectaffect only the container application. In OLE linking, the Word documentof the example is the container application and the Excel spreadsheet isa linked object. The container application contains a pointer to thelinked object and any changes made to the linked object change theoriginal linked object. Any other applications that link to the linkedobject are also updated. POLESS supports structured storage such thatonly one change made to an object stored in exam resource file 120 isglobally effective. Test driver 110 comprises Active Document containerapplication 112 for the visible plugins, display plugin 152, helm plugin154, and item plugin 156, which function as embedded objects, preferablyCOM objects.

Both XXL compiler 140 and plugins 150 are involved in storing thecompiled test content into exam resource file 120, if any of plugins 150are being used. Exam resource file 120 comprises, for example, ahierarchical storage structure, as will be described in further detailbelow. Other storage structures may optionally be used. XXL compiler 140determines to which storage location a specific segment of the compiledtest content is to be stored. However, if any of plugins 150 are used tovalidate the portion of any of the data from exam source files 130, thenthe plugins 150 store the data directly to the exam resource file, basedupon directions from XXL compiler 140. XXL compiler usesIPersistResource interface 192, co-located with I-Plugin interface 167in FIG. 3, to control the persistence of the data to exam resource file120. XXL compiler 140 and plugins 150 write the data to exam resourcefile 120 using POLESS interfaces 191.

FIG. 5 illustrates the contents of exam source file 130, which arecompiled into exam resource file 120 by XXL compiler 140 and plugins150. FrontPage 2000 Web 200 is used, for example, to author the test.Exam source files 130 contain media files 210, visual files 220, andlogic files 230. Media files 210 are multimedia files used to enhancethe presentation of the test, including, for example, XML data files212, sound files 214, image files 216, and binary files 218. XML datafiles 212 include the XXL test definition language and the XXLextensions from the plugins 150 that use XML. The test specification,presentation, scoring and other information is specified in the XMLfiles. Sound files 214 include any sounds that are to be used during thetest, such as .mp3 files, .au files, etc. Image files 216 include anyimages to be used during the test, such as .jpg files, gif files, etc.Binary files 218 include any data needed by a plugin 150 that is not inXXL format. Visual files 220 are HTML files that specify the visualpresentation of the test as presented to the examine on the displaydevice, including items files 222, presentation files 224, score reportfiles 226, and custom look files 228. Items files 222 include HTML filesthat are used to specify the visual component of test questions, e.g.,stems and distractors. Items files 222 are capable also of referencingexternal exhibits. An exhibit could be a chart, diagram or photograph.Formats of exhibits include, for example: .jpg, .png, etc. Presentationfiles 224 define what is seen by the candidate on the display device ata particular instant during the test. Score report files 226 istypically an HTML file with embedded script that includes, for examplecandidate demographics, appointment information, and candidateperformance. The performance might include pass/fail, achievement indifferent content areas, etc. Custom look files 228 are typically HTMLfiles with embedded script to layout, for example, the title bar andinformation contained therein. Logic files 230 are XML files thatspecify the functional aspects of the test, including test specificationfiles 232, plugin files 234, item bank files 236, and template files238. Test specification files 232 specify the content and progress ofthe test as provided by the client. Plugin files 234 define plugins 150and contain any data necessary to implement plugins 150. Item bank files236 include the data content and properties of the items, or testquestions, that are to be presented to the candidate during the test.Properties of an item include the correct answer for the item, theweight given to the item, etc. Template files 238 define visual layoutsthat are used with the display screen during the test.

Referring again to FIG. 3, once a test has begun, test driver 110accesses exam resource file 120 for the instructions and files needed toimplement the test, using POLESS interfaces 193. Test driver 110 alsoaccesses plugins 150 for additional data that expands the functionalityof test driver 110 in the areas of items, navigation algorithms,information displays, scoring algorithms, timing algorithms, test unitselection algorithms, results persistence reporting, printed scorereporting, and/or helm types. Test driver 110 communicates with plugins150 using various COM interfaces 169. COM interfaces facilitate OLElinking. As stated previously, test driver 110 is an Active Documentcontainer application and plugins 150 are embedded objects. The COMinterfaces function as communications paths between the containerapplication and the objects.

There are, for example, ten COM interfaces utilized in computer-basedtest delivery system 100. IPlugin interface 167, which is also a COMinterface, is supported by all of plugins 150. COM interfaces 169,therefore, includes the IPlugin interface. The IPlugin interfacecontains generic operations such as loading and unloading, required ofall plugins 150. In addition to the global IPlugin interface, eachplugin 150 also uses, for example, a second, individual COM interface169 to communicate with test driver 110. Alternative structures of theIPlugin interface may also be used. Table 1 shows the relationshipbetween each plugin 150 and the COM interface 169 used with thatparticular plugin 150. TABLE 1 COM INTERFACE FOR PLUGINS. PLUGIN COMINTERFACE DESCRIPTION All Plugins 150 IPlugin Passes data between thetest driver and all plugins regarding generic operations, e.g., loadingand unloading. Display 152 IDisplay Passes data between the test driverand the visible plugins that handle title bars, displays, non- answereditems, and summaries. Helm 154 IHelm Passes data between the test driverand the visible plugins that display navigation controls or reviews.Communicates with a navigation plugin to perform the actual navigation.Also functions as a user interface connection to the test driver. Item156 IItem Passes data between the test driver and the visible pluginsthat govern test items or simulations. Timer 158 IUnitTimer Passes databetween the test drivers and the invisible plugins used to performtiming across examination sections. Selection 160 ISelection Passes databetween the test drivers and the invisible plugins used to select forms,sections, groups, or items for delivery to the candidate. Navigation 160INavigate Passes data between the test drivers and the invisible pluginsused to control section navigation and define rules for traversingthrough the test. Scoring 164 IScore Passes data between the testdrivers and the invisible plugins used to control scoring of deliveredtesting units. Results 166 IResults Passes data between the test driversand the invisible plugins that control writing of candidate results, forexample, to candidate exam results file 180. Report 168 IReport Passesdata between the test drivers and the invisible plugins that controlprinting of score reports and other material, for example, printedreference material and post exam instructions to printer 182.

Exam instance file 170 is used to restart a test if the test has beeninterrupted, for example, because of a power failure. During delivery ofthe test, exam instance file 170 receives examination state informationfrom test driver 110 and plugins 150 regarding the state of all runningobjects being used to deliver the test. The examination stateinformation includes the presentation that was being delivered on thedisplay device before the interruption, the responses the candidate hadentered in that presentation, etc. When the test is restarted, the examinstance file 170 loads the state information back to test driver 110and plugins 150, allowing the test to return to operation at the pointwhere the test had been interrupted. Preferably, the running state ofall objects is saved to exam instance file 170 rather than of only someof the objects. Saving the state of only some of the objects to examinstance file 170 causes the potential problem of only a portion of thetest information being restored after a test interruption. Exam instancefile 170 may also store additional information relating to the test,including, for example: the timing utilized and time remaining on unitsof the exam, the current unit of delivery, candidate score, etc. Testdriver 110 and plugins 150 communicate with exam instance file 170 usingPOLESS interfaces 195. Test driver 110 controls communications betweentest driver 110 and plugins 150 using IPersistInstance interface 196,which is collocated with COM interfaces 169 in FIG. 3.

Several administrative environments perform the administrative functionsof computer-based test delivery system 100, for example: Test CenterManager (“TCM”) Bridge 172; Educational Testing Service (“ETS”) Bridge174; and Unified Administration System (“UAS”) 174. Administrativefunctions include, for example: checking-in an candidate, starting thetest, aborting the test, pausing the test, resuming the test, andtransmitting results.

There are preferably two ways to run test driver 110. The first isthrough a series of command line options and the second is using COMinterfaces describing appointment information. The command line optionexists for backwards compatibility in a standard ETS environment and aTCM environment. Table 2 shows a list of command line options testdriver 110 supports. There are, for example, four programs which launchthe test through the COM interface, for example: 1) LaunchTest.exe (fortest production and client review); 2) UAS; 3) UTD2ETS.dll (an internalcompatibility module for use with the ETS administration environment);and 4) UTD2TCM (for the Test Center Manger environment). Other number ofenvironments and/or programs may optionally be used. TABLE 2 COMMANDLINE OPTIONS SUPPORT BY TEST DRIVER. SWITCH (ES) OPTION(S) PURPOSE /?n/a Displays command line /help switches in dialog box. /UnregServer n/aUnregisters the test driver core COM server. /RegServer n/a Registersthe test driver core COM server. /T Form name Name of the form or formgroup to run in the exam. /F Resource file The exam resource file touse. /S n/a Suppress any printing. /W n/a Run in close-of-day mode. /TIn/a Set tracing level to information. (Very large instance file). /TWn/a Set tracing level to warning. (Large instance file.) /TE n/a Settracing level to error. (Average sized instance file.) /K Resource dir,Used to point to SKSID, candidate directories. A space directorseparates each of the three options.

The administration environments use several interfaces to communicatewith test driver 110. IAppointment interface 176 is part of UAS 174 andallows access by test driver 110 to candidate information for thecandidate taking the test, such as demographics. The candidateinformation is included in candidate exam results file 180, which iscreated by the test driver. ILaunch2 interface 177 functions as theprimary control interface for UAS 174 and allows UAS 174 to controlvarious components such as test driver 110, screen resolution change,accommodations for disabled candidates, candidate check-in, etc., in atest center, which is the physical location where the candidate istaking the test. ITransfer interface 199 transfers candidate examresults file 180 and other files back to UAS 174. IPrint interface 198sends information regarding any reports to printer 182.

The test driver is described in greater detail in U.S. PatentPublication No. 20030182602, entitled “METHOD AND SYSTEM FOR COMPUTERBASED TESTING USING PLUGINS TO EXPAND FUNCTIONALITY OF A TEST DRIVER”,incorporated herein by reference.

II. XXL Compiler Interfaces and Classes

FIGS. 6A and 6B illustrate the main diagram for XXL compiler 140. XXLcompiler 140 comprises the following classes, for example: cCompile2000; cData 2004; cArea 2006; cTemplate 2008; cCategory 2010; cItem2012; cPresentation 2014; cGroup 2016; cSection 2018; cForm 2020;cFromGroup 2022; cExam 2024; cMsgBox 2026; cChecksum 2028; cEvent 2030;cResult 2032; cReport 2024; cPlugin 2036; and cXXL 2038.

The main interface to XXL compiler 140 is ICompile interface 2002.ICompile interface 2002 is implemented by cCompiler class 2000. Allcontrol and initiation of compilation of exam source files 130 into examresource file 120 occurs by way of this single public interface. Thecore, non-plugin related elements of the XXL test definition language,as stored in XXL files 134, are compiled by classes in XXL compiler 140.For example, cSection class 2018, compiles the section element, andcGroup class 2016 compiles the group element.

ICompile interface 2002 supports the following operations, for example:createResource( ); addSource( ); addData( ); closeResource( ); about( );linkResource( ); openResource( ) and getCryptoObject( ). CreateResource() creates a resource file, for example, an XXL based resource file suchas exam resource file 120. AddSource( ) compiles an XXL file into theresource file. AddData( ) adds a file directly to a data branch of theresource file. CloseResource( ) closes the resource file. LinkResource() links a resource in the resource file and is performed after allcompiling of the source files are completed. GetCryptoObject( ) returnsan ICrypto object containing the current encryption setting of POLESS,as described below.

The classes of XXL compiler 1040, e.g., cForm 2020 and cItem 2012,handle individual XXL core language elements. All of these classescompile the specific XXL source element into exam resource file 120. Allof these class language elements are also symbols used in laterreferences. Therefore, the classes all derive from cSymbol class 2040.cSymbol class 2040 allows the classes of XXL compiler 140 to reside in asymbol table.

For example, the XXL element plugin 150 appears as follows in XXL files134: 3     <plugin     name=“helmNextPrevious”progid=“UTDP.cNextPrevious” />

This XXL call causes an instance of cPlugin class 2036 to be created,compiles the source, and writes the compiled result to exam resourcefile 120. The name and ID of Plugin 150 is also added to the symboltable for later reference.

XXL compiler 140 also contains the following token classes, for example:cToken 2042; cTokenCreator NoRef 2044; cTokenCreator 2046;ctokenCreatorRef 2048; cTokenCreatorBase 2050; and cTokenFactory 2054.These token classes are involved in the identification of tokens. Tokensturn into symbols after identification. Symbols are any class derivedfrom cSymbol, e.g., cTemplate, cSection, etc.

XXL compiler 140 also contains the following symbol table classes, forexample: cPluginSymbolTable 2058; cTemplateSymbolTable 2060;cSymbolTable 2062; cFFGSymbolTable 2064; cSGPSymbolTable 2066; andcSymbolTableBase 2068. These classes are varieties of symbol tables.There are different symbol tables for different groups of symbols. Agroup of symbols define a name space for the symbol. Common symbol tablefunctions are located in the base symbol table classes and templates.

All content and specification destined for a plugin 150 appears in thedata element in XXL. For example, below is an item definition in XXL: 4 <item  name=“wantABreak1” skipAllowed=“false”> <data> <multiChoice  correctAnswer=“A”    maxResponses=“1”minResponses=“1”            autoPrompt=“false”URI=“itembank/info_item.htm#wantABreak”/>- ; </data> </item>

The item element is handled by a cItem class 2012 object. The dataelement in the XXL definition is handled by a cData class 2004 object.Item plugin 156 Plugin 150 will receive the source to compile from thecData class 2004 object, in this example, a multiChoice element.

-   -   cWrapXML class 2052, a wrapper class for XML DOM nodes, supports        error handling. cCustomAttributes class 2056 compiles the custom        attributes XXL element. cWrapPropertySet class 2070 is a wrapper        class for a POLESS property storage.

III. Test Driver Interfaces and Classes

A. Interfaces

FIG. 7 shows test driver 110, UAS 174, and the interfaces used by andbetween the test driver 110 and UAS 174 to deliver the test. UAS 174defines ILaunch2 interface 177, which is used by UAS 174 to initiatetest events. ILaunch2 interface 177 is an extension of ILaunch interface178, which, in other embodiments of the present invention, is also usedby UAS 174 to initiate test events. UAS 174 also defines and implementsadditional interfaces, for example: IAppointment interface 176; IPrintinterface 198; and ITransfer interface 199. IAppointment interface 176transfers examinee candidate information and appointment details fromUAS 174 to test driver 110, as is illustrated by the dashed arrowconnecting IAppointment interface 176 to test driver 110. IPrintinterface 198 allows UAS 174 to send print requests to printer 198regarding reports, for example, score reports. ITransfer interface 199allows UAS 174 to request the transfer of information from candidateexam results file 180 back to UAS 174.

Test driver 110 defines various interfaces to allow test driver 110 tocommunicate with different parts of computer-based test delivery system100. Test driver 110 includes, for example, ten COM interfaces 169 tocommunicate and transfer data with plugins 150. (See Table 1 above) TheCOM interfaces 169 are denoted in FIG. 7 as follows, for example:IDisplay interface 169 a; IHelm interface 169 b; IItem interface 169 c;IUnitTimer interface 169 d; ISelection interface 169 e; INavigateinterface 169 f; IScore interface 169 g; IResults interface 169 h;IReport interface 169 i; and IPlugin interface 169 j.

Test driver 110 and plugins 150 communicate and transfer data with examresource file 120 using, for example, three IPersistResource interfaces192: IPersistResourceStream interface 192 a; IPersistResourceSetinterface 192 b; and IPersistResourceStore interface 192.IPersistResource interfaces 192 are used by plugins 150 duringcompilation of exam source files 130 and are used by both test driver110 and plugins 150 during delivery of the test. During compilation ofexam source files 130, XXL compiler 140 directs plugins 150 in whichstorage location of exam resource file 120 to store any information thatplugins 150 have validated. Plugins 150 can then retrieve the storedinformation from exam resource file 150 during delivery of the test.Other number of interfaces and different combination of functionalitymay alternatively be used.

Information is saved from plugins 150, or from XXL compiler 140 ingeneral, to exam resource file 120, for example, as either a stream ofdata, as a set of data, or as a storage structure, depending on which ofthe three IPersistResource interfaces 192 is implemented to save theinformation from plugins 150, to exam resource file 120.IPersistResourceStream interface 192 a saves the information, forexample, as a stream of data or other data storage format. A stream ofdata is simply a stream of bytes stored as a linear sequence.IPersistResourceSet interface 192 b saves the information, for example,as a set of data. A set of data is preferably a name-value propertypair. For example, the name of a particular property for an item isdistractors and the value is the number of distractors required for thatitem. IPersistResourceSet interface 192 allows the name-value propertypair to be saved together in exam resource file 120.IPersistResourceStore interface 192 c saves the information, forexample, in a directory format with storage areas. The directory formatallows other streams of data to be saved within the storage area, otherproperty sets to be stored within the storage area, and for sub-storagesto be saved under the storage area.

IPersistInstance interface 196, likewise, comprises, for example, three,different interfaces, for example: IPersistInstanceStream interface 196a; IPersistInstanceSet interface 196 b; and IPersistInstanceStoreinterface 196 c. Examination state information is saved to exam instancefile 170 as, for example, a stream of data, as a set of data, or as astorage element, depending on which of the three IPersistResourceinterfaces 192 is implemented.

Two of the interfaces, IContainerNotify interface 200 andIContainerNotifyHelm interface 206, function as callback interfaces fromplugins 150 to test driver 110. IContainerNotify interface 200 allows avisible plugin to inform test driver 110, for example, that the pluginis displayed and ready for examinee interaction. IContainerNotifyHelminterface 206 allows helm plugin 154 to request navigation from testdriver 110 after receiving an input from the examinee to move to anothersection of the test. IMore interface 202 is used to convey whether theexaminee has seen all content in a presentation. For example, a “more”button appears in place of the next button when the content exceeds thewindow length. When the examinee scrolls to the bottom, the “more”button disappears and is replaced with the “next” button. Collectioninterface 204 is used by test driver 110 to hold any group entities, forexample, categories and sections of the test.

The remaining interfaces are, for example, Microsoft defined ActiveDocument interfaces, used to implement OLE linking functions of testdriver 110 and the visible plugins, display plugin 152, helm plugin 154,and item plugin 156. IOleInPlaceFrame interface 210 controls thecontainer's top-level frame window, which involves allowing thecontainer to insert its menu group into the composite menu, install thecomposite menu into the appropriate window frame, and remove thecontainer's menu elements from the composite menu. IOleInPlaceFrameinterface 210 sets and displays status text relevant to the end-placeobject. IOleInPlaceFrame interface 210 also enables or disables theframes modeless dialogue boxes, and translates accelerator key strokesintended for the container's frame. IOleInPlaceUI window interface 211is implemented by container applications and used by object applicationsto negotiate boarder space on the document or frame window. Thecontainer provides a RECT structure in which the object can placetoolbars and other similar controls, determine if tools can in fact beinstalled around the objects' window frame, allocates space for theboarder, and establishes a communication channel between the object andeach frame and document window. IAdviseSync interface 212 enablescontainers and other objects to receive notifications of data changes,view changes, and compound-document changes occurring in objects ofinterest. Container applications, for example, require suchnotifications to keep cached presentations of their linked and embeddedobjects up-to-date.

Calls to IAdviseSync interface 212 methods are a synchronous, so thecall is sent and then the next instruction is executed without waitingfor the calls return. IOleWindow interface 213 provides methods thatallow an application to obtain the handle to the various windows thatparticipate in-place activation, and also to enter and exitcontext-sensitive help mode. IOleInPlaceSite interface 214 managesinteraction between the container and the objects in-place client site.The client site is the display site for embedded objects, and providesposition and conceptual information about the object. IOleClientSiteinterface 215 is the primary means by which an embedded object obtainsinformation about the location and extent of its display site, itsmoniker, its user interface, and other resources provided by itscontainer. Test driver 110 called IOleClientSite interface 215 torequest services from the container. A container must provide oneinstance of IOleClientSite interface 215 for every compound-document itcontains. IOleDocumentSite interface 216 enables a document that hasbeen implemented as a document object to bypass the normal activationsequence for in-place-active objects and to directly instruct its clientsite to activate it as a document object. A client site with thisability is called a “document site”.

B. Core Classes

FIGS. 8A and 8B illustrate the main classes of test driver 110 and theinterfaces between test driver 110 and plugins 150. Also shown are theclasses that interface to UAS 174. ITransfer interface 199, IPrintinterface 198, ILaunch2 interface 177, and IAppointment interface 176represent the connections from test driver 110 to UAS 174, as describedpreviously. Some of the lines depicted in FIG. 8 are solid and some aredashed. The solid lines, for example, between IcResults interface 240and cEvent class 252, represent inheritance. The dashed lines, forexample, between IExam interface 222 and IPlugin interface 169 j,represent instantiation.

Inheritance, or generalization, relates to a generalized relationshipbetween classes that shows that the subclass shares the structure orbehavior defined in one or more superclasses. A generalized relationshipis a solid line with an arrowhead pointing to the superclass.Instantiation, or dependency, represents a relationship between twoclasses, or between a class and an interface, to show that the clientclass depends on the supplier class/interface to provide certainservices. The arrowhead points to the supplier class/interface. Someservices from a supplier class to a client class include, for example:the client class access a value (constant or variable) defined in thesupplier class/interface; methods of the line class invoke methods ofthe supplier class/interface; and methods of the client class havesignatures whose return class or arguments are instances of the supplierclass/interface. For instantiation, the cardinality of the relationshipis illustrated in FIG. 8 if the relationship represents containment.Cardinality specifies how many instances of one class may be associatedwith a single instance of another class. Cardinality can be shown forrelationships to indicate the number of links allowed between oneinstance of a class and the instances of another class.

Test driver 110 also has several interfaces and implementing classes.Test driver 110 interfaces include, for example: IExam interface 222;IMsgBox interface 224; ICategory interface 232; IForm interface 238;IcResults interface 240; IcReport interface 242; IScript interface 246;ISection interface 250; IPresentation interface 248; and/or IcIteminterface 256. The classes that implement the main interfaces include,for example: cScreenMinimum class 226; cFormGroup class 228; cPluginclass 230; cArea class 234; cTemplate class 236; cActivePlugin class250; and cEvent class 252. The interfaces that are prefaced by “Ic” havenames that already exist for plugins 150 to enact, for example, itemplugin 156 implements IItem interface 169 c. IcItem interface 256,however, is the interface implemented by test driver 110 class citem(not shown). Of course, any number of interfaces may be used, dependingon the necessary functionality.

The core class cExam (not shown) implements ILaunch2 interface 177 sothat UAS 174 can control test driver 110. The appointment object, whichimplements IAppointment interface 176, is the main object UAS 174supplies to test driver 110. The appointment object is available toplugins 150 by way of IPlugin interface 169 j. Furthermore, all plugins150 get (IExam) using the IPlugin interface 169, also.

The cExam class selects and delivers the form, using cFormGroup class228 and IForm interface 238. The form delivers results using IcResultsinterface 240, reports using IcReport interface 242, and sectionscontained with in the test using ISection interface 250. Classes thatare in the test delivery chain preferably derive from cEvent class 252.

The cResults class (not shown) delivers a results plugin 166 thatimplements IResult interface 169 i. The cReport class (not shown)delivers a report plugin 168 that implements IReport interface 169 h.The cSection, cGroup, and cForm classes (not shown) use severalinvisible plugins 150 to control the delivery of the test. These plugins150 are timer plugins 158, which implement IUnitTimer interface 169 d,selection plugins 160, which implement ISelection interface 169 e,scoring plugins 164, which implement IScore interface 169 g, andnavigation plugins 162, which implement INavigate interface 169 f. ThecPresentation class (not shown) supplies data to its template for thedisplay of the presentation. The three visible plugins 150 are createdand controlled through cTemplate class 236 and child objects cArea class234. Item plugins 156 have an extension class in the cItem class (notshown) that wraps the item plugin 156 and provides generic extendedservices that all item plugins 156 implements. The cItem class in testdriver 110 is a wrapper class. The cItem class provides two baseservices, for example: generic item functionality and access to itemplugin 156, which is the wrapping function. Item generic functionalityincludes, for example: having an item name, having an item title,determining if the item is scored or un-scored, determining whether theitem has been presented to the examinee, etc. These services are genericto all items and are provided by test driver 110. Item plugins 156perform the actual scoring of the item, which is unique to each itemtype. Item plugins 156 present the content of the item and allow theexaminee to interact with the item. These services are unique to eachitem type.

In addition to the interfaces described previously, test driver 110implements IRegistry interface 220, which allows VB code to access theWindows registry. Test driver 110 also implements ILegacyItem interface258 and ILegacyScore interface 260, which are defined by test driver 110and are implements by certain item plugins 156 and scoring plugins 164.ILegacyItem interface 258 and ILegacyScore interface 260 allow old itemtypes that existed in previous test drivers to report results like theprevious test drivers. For some tests, test driver 110 must reportresults for old item types, which had very specific ways of reportingresults. ILegacyItem interface 258 and ILegacyScore interface 260 allowthe new item plugins 156 that represent old item types to report thislegacy format of information to result plugins 166 trying to imitateprevious test drivers.

A complete description of test driver 110 classes and interfaces isincluded in Appendix A.

IV. POLESS

All persistent storages, exam resource file 120 and exam instance file170, preferably utilize POLESS. POLESS allows data to be embedded,linked, or references as external files from the persistent storage totest driver 110 and Active Document container application 112 (FIG. 3).POLESS supports a hierarchical tree structure with node or branch leveladditions, replacements, and deletions. POLESS also supports optionaldata encryption at the node level. The type of encryption employeddepends on the destination of the information in the persistent storage.For example, different encryption keys may optionally be used for databeing routed to test centers, data being routed to administrative datacenters, and data being routed for client use (e.g., client review).Microsoft Crypto-API is preferably used to perform encryption of data inthe persistent storage. Finally, POLESS also supports optionalcompression at the node level, preferably using Lempal-Zev compression.

POLESS is an extension of OLE structured storage compound documentimplementation. A compound document is a single document that contains acombination of data structures such as text, graphics, spreadsheets,sound and video clips. The document may embed the additional data typesor reference external files by pointers of some kind. There are severalbenefits to structured storage. Structured storage provides file anddata persistence by treating a single file as a structured collection ofobjects known as storage elements and streams. Another benefit isincremental access. If test driver 110 or plugins 150 need access to anobject within a compound file, only that particular object need beloaded and saved, rather than the entire file. Additionally, structurestorage supports transaction processing. Test driver 110 or plugins 150can read or write to compound files in transacted mode, where changesmade can subsequently be committed or reverted.

A. POLESS Components

FIG. 9 shows the major components that support POLESS and the interfacesthat connect the components. POLESS 300 may be either exam resource file120 or exam instance file 170. POLESS 300 utilizes PKware librarycomponent 330 for storage compression and decompression. POLESS 300 usesCrypto API component 332, a Microsoft application, for storageencryption and decryption. Crypto API component 332 relies on a cryptoservice provided (“CSP”) 334 to perform the actual encryptionalgorithms. Access to the services of these components is facilitated bystandard (API) interfaces exposed by these components.

OLE2SS component 310 contains all the interface definition that makeupstructure storage. These interfaces can be realized by any structuredstorage implementation, such as compound document implementation OLE2320 and POLESS 300. The interfaces include, for example: IStreaminterface 340; ISequentialStream interface 342; IStorage interface 344;and IRootstorage interface 346. POLESS 300 additionally implementsIStreamVB interface 348 and IStorageVB interface 350.

IStreamVB interface 348 supports several functions, for example: ReadVB(); WriteVB( ); Clear( ); Reset( ); get_sName( ); get_oStream( ); andCopyTo( ). ReadVB( ) reads a specified number of bytes to a data array.WriteVB( ) writes the byte data to the stream. Clear( ) clears thestream of all data. Reset( ) sets position to the beginning of thestream. get_sName( ) is a read-only function that returns the name ofthe stream. get_oStream( ) is a read-only function that returns theIStream interface 348. CopyTo( ) copies a source stream to a destinationstream.

IStorageVB interface 350 supports several functions, for example: Clear(); CommitVB( ); RevertVB( ); sElementName( ); bStorage( ); oElement( );CreateStream( ); OpenStream( ); CreateStorage( ); OpenStorage( );get_sName( ); get_oStorage( ); get_nCount( ); GetCompression( );GetEncryption( ); GetCRC( ); CreateStreamLinked( ); CreatePropertyStg(); OpenPropertyStg( ); SetClass( ); RegisterAlias( ); Destroy( ); andget_ElementType( ). Clear( ) clears the storage of all elements.CommitVB( ) causes transacted mode changes to be reflected in theparent. RevertVB( ) discards changes made since the last commit.sElementName( ) returns the name of the element. bStorage( ) returnsTRUE if the element is a sub-storage. oElement( ) returns IStreamVBinterface 348 or IStorage interface VB 350 for the element.CreateStream( ) creates and opens a stream and returns IStreamVBinterface 348.

OpenStream( ) opens a stream and returns IStreamVB interface 348.CreateStorage( ) creates and opens a nested storage and returnsIStreamVB interface 348. OpenStorage( ) opens an existing storage andreturns IStreamVB interface 348. get_sName( ) is a read-only functionthat returns the name of the storage. get_oStorage( ) is a read-onlyfunction that returns IStorage interface 350. get_nCount( ) is aread-only function that returns a count of the elements. GetCompression() returns the status of file compression. GetEncryption( ) returns thestatus of file encryption. GetCRC( ) returns the status of file CRCchecking. CreateStreamLinked( ) creates and opens a linked stream andreturns IStreamVB interface 348. CreatePropertyStg( ) creates and opensa property storage and returns IPropertyStorageVB interface 414.OpenPropertyStg( ) opens a property storage and returnsIPropertyStorageVB interface 414. SetClass( ) sets the CLSID for thestorage. RegisterAlias( ) registers a pluggable protocol. Destroy( )destroys the specified elements. get_ElementType( ) is a read-onlyfunction that returns the type of the element.

B. POLESS Classes

FIGS. 10A and 10B illustrate the main class of POLESS 300, theinterfaces used to implement the classes, and the flow of the creationof streams 424 and storages 426. cFileRoot class 400 is the first objectinstantiated and is used to create a new or open an existing a POLESSfile. cStorageRoot class 406 is returned, which is a slightly overloadedversion of cStorage class 410. From cStorageRoot class 406 creates oropens cStream class 408 and cStorage class 410, from which any streamsor storages and sub-storages of those can be created or opened,respectively. For instance, cStorage class 410 creates cPropertyStorageclass 412, which creates storage for property sets. The classesimplement interfaces that perform operations and/or define attributesthat further define the function or properties of the class. A completedescription of POLESS 300 classes and interfaces is included in AppendixB.

1) cFileRoot Class

-   -   cFileRoot class 400 is the root POLESS class and controls the        creation and opening of all POLESS files. cFileRoot class 400 is        generally instantiated first before any other POLESS objects can        be created, although other sequences are possible. cFileRoot        class 400 implements IFileRoot interface 401, which is        collocated in FIG. 10 with cFileRoot class 400. IFileRoot        interface 401 is used to open one file at a time and is not        released until all other storage object 426, stream object 424,        and property storage interfaces are released and the file is        ready to be closed. cFileRoot class 400 and IRoot interface        support the following operations, for example:        StorageFileCreate( ); StorageFileOpen( ); CryptoGet( );        bStorageFile( ); StorageAmalgamatedGet( ); DeltaFileCreate( );        DeltaFileApply( ); GetObjectFromPath( ); CreateStreamFromBSTR(        ); MemoryStreamFromStream( ); GetPicture( ); and SavePicture( ).

StorageFileCreate( ) creates a new storage file, returns the rootstorage to interface, marks the new structured storage file as a POLESSfile by storing the class ID (“CLSID”) of this class in a stream in theroot storage. StorageFileOpen( ) opens an existing storage file andreturns the root storage interface. CryptoGet( ) gets a defaultconfigured crypto class and should be set and used on the open or createof the storage file. bStorageFile( ) returns true if the file providedis an OLLE structured storage file and not a POLESS storage file.StorageAmalgamatedGet( ) gets an empty small cStorageAmalgamated class404. DeltaFileCreate( ) creates a POLESS difference file by comparingthe original POLESS file to the updated POLESS file. DeltaFileApply( )applies a POLESS delta file and applies the original POLESS file to thedelta file to create an updated POLESS file. GetObjectFromPath( ) usesmonikers to retrieve the object named by the path and returns a pointerto the object retrieved. CreateStreamFromFile( ) creates a structuredstorage stream and populates it with the contents of the file.CreateStreamFromBSTR( ) creates a structures storage stream and fills itwith the specified string. MemoryStreamFromStream( ) is used to copy astream to a newly created memory stream object. GetPicture( ) loads apicture from stream object 424. SavePicture( ) saves the picture intothe stream 426.

2) cCrypto Class

-   -   cCrypto class 402 controls the configuration of the        encryption/decryption of POLESS 300. cCrypto class 402 has the        following attributes, for example: sProviderName; eProviderType;        sContainerName; and sPassword. SProviderName represents the name        of CSP 334 being used to perform the encryption/decryption        services. eProviderType is the type of CSP 334. The field of        cryptography is large and growing. There are many different        standard data formats and protocols. These are generally        organized into groups or families, each of which has its own set        of data formats and way of doing things. Even if two families        used the same algorithm, for example, the RC2 block cipher, they        would often use different padding schemes, different key links,        and different default modes. Crypto API is designed so that a        CSP provider type represents a particular family. sContainerName        is the key name and must be provided by cCrypto class 402.        sPassword is an optional password on the public/private key pair        and can only be entered by a human operator. The password can be        used for review disks and their resource files.

cCrypto class 402 implements ICrypto interface 401 and they support thefollowing properties and method, for example: ProviderName; Password;FileType; Algorithm; EnumProviders( ); and EnumAlgorithms( ).Get_ProviderName( ) returns the name of the Crypto provider.Put_ProviderName( ) sets the name of the Crypto provider. Get_Password() and Put_Password( ) are only used for sponsor resource files.Get_FileType( ) gets the file type and put_FileType( ) sets the filetype. Get_Algorithm( ) gets the encryption algorithm and put Algorithm() sets the encryption algorithm. EnumProviders( ) returns an enumeratorfor the list of installed providers. EnumAlgorithms( ) enumerate a listof algorithms for the current provider.

-   -   3) cStorageAmalgamated Class    -   cStorageAmalgamated class 404 is an implementation of IStorage        interface 344. cStorageAmalgamated class 404 holds references to        an ordered collection of IStorage objects. When a stream is        opened, cStorageAmagalmated class 404 searches the collection of        storage objects in order to find the first storage object that        has the requested stream and returns this stream.        cStorageAmalgamated class 404 handles compound storage        resolution and delegates all other work to cStorage class 410.        cStorageAmalgamated class 404 is, for example, read-only.        cStorageAmalgamated class 404 will not allow stream or storages        to be created but is primarily for reading exam resource file        120. cStorageAmalgamated class 404 implements        IStorageAmalgamated interface 405. cStorageAmalgamated class 404        and IStorageAmalgamated interface 405 support the following        operations, for example: StorageAdd( ); ClearStorage( );        OpenStorageAmalgamated( ); and OpenPropertyStgAmalgamated( ).        StorageAdd( ) adds a new storage to the collection of storages.        ClearStorage( ) clears all the storage objects from the        collection. OpenStorageAmalgamated( ) opens a sub-storage of the        current amalgamated storages in an amalgamated fashion.        OpenPropertyStgAmalgamated( ) opens a property storage of the        current amalgamated storages in an amalgamated fashion.        Amalgamation is described in greater detail, in U.S. Patent        Publication No. 20030129573, entitled “EXTENSIBLE EXAM LANGUAGE        (XXL) PROTOCOL FOR COMPUTER BASED TESTING,” incorporated herein        by reference.

4) cStorageRoot Class

-   -   cStorageRoot class 406 is the POLESS implementation of IStorage        interface 344 and IRootstorage interface 346. cStorageRoot class        406 handles any storage object 426 that is POLESS specific and        then delegates work to the cStorage class 410. IRootstorage        interface 346 supports the SwitchToFile( ) operation, which        copies the current file associated with the storage object to a        new file, which is then used for the storage object and any        uncommitted changes. cStorageRoot class 406 also implements        IPersistFile interface 418, which provides methods that permit        an object to be loaded from or saved to a disk file, rather than        a storage object or stream. Because the information needed to        open a file varies greatly from one application to another, the        implementation of IPersistFile::Load on the object preferably        also open its disk file. IPersistFile interface 418 inherits its        definition from IPersist, so all implementations must also        include the GetClassID( ) method of IPersist interface 418.

5) cStream Class

cStream class 408 is the POLESS implementation of IStream interface 340.cStream class 408 handles any storage object 426 that is POLESS specificand then delegates work to compound document implementation OLE2 320.The specific work includes compression/decompression andencryption/decryption of stream object 424.

IStream interface 340 supports the following operations, for example:Seek( ); SetSize( ); CopyTo( ); Commit( ); Revert( ); LockRegion( );UnlockRegion( ); Stat( ); and Clone( ). Seek( ) changes the seek pointerto a new location relative to the beginning of stream object 424, theend of stream object 424, or the current seek pointer. SetSize( )changes the size of stream object 424. CopyTo( ) Copies a specifiednumber of bytes from the current seek pointer in stream object 424 tothe current seek pointer in another stream object 424. Commit( ) ensuresthat any changes made to a stream object 424 open in transacted mode arereflected in the parent storage object. Revert( ) discards all changesthat have been made to a transacted stream since the last call toIStream::Commit. LockRegion( ) restricts access to a specified range ofbytes in stream object 424. Supporting this functionality is optionalsince some file systems do not provide this operation. UnlockRegion( )removes the access restriction on a range of bytes previously restrictedwith IStream::LockRegion. Stat( ) retrieves the STATSTG structure forthe stream object 424. Clone( ) creates a new stream object thatreferences the same bytes as the original stream but provides a separateseek pointer to those bytes.

IStreamVB interface 348 is an automation friendly version of IStreaminterface 340. IStreamVB interface 348 supports the followingoperations, for example: Read( ); Write( ); Clear( ); Reset( );get_sName( ); get_oStream; and CopyTo( ). Read( ) reads data from streamobject 424. Write( ) writes data, including the entire byte array, tostream object 424. Clear( ) clears stream object 424 of all data. Reset() resets the position in stream object 424 to the beginning of streamobject 424. Get_sName( ) returns the name of the stream. Get_Stream( )returns the IDispatch interface. CopyTo( ) copies the contents of asource stream to a destination stream.

6) cStorage Class

cStorage class 410 is the POLESS implementation of IStorage interface344 and IcStorage interface 411. cStorage class 410 handles any storageobject 426 that is POLESS specific and then delegates work to compounddocument implementation OLE2 320.

IStorage interface 344 supports the following operations, for example:CreateStream( ); OpenStream( ); CreateStorage( ); OpenStorage( );CopyTo( ); MoveElementTo( ); Commit( ); Revert( ); EnumElements( );DestroyElement( ); RenameElement( ); SetElementTimes( ); SetClass( );SetStateBits( ); and Stat( ). CreateStream( ) creates and opens a streamobject 424 with the specified name contained in a storage object.OpenStream( ) opens an existing stream object 424 within a storageobject using specified access permissions. CreateStorage( ) creates andopens a new stream object 424 within a storage object. OpenStorage( )opens an existing storage object 426 with the specified name accordingto the specified access mode. CopyTo( ) copies the entire contents of anopen storage object 426 into another storage object. The layout of thedestination storage object may differ from the layout of the sourcestorage object. MoveElementTo( ) copies or moves a sub-storage or streamobject 424 from one storage object 426 to another storage object.

Commit( ) reflects changes for a transacted storage object 426 to theparent level. Revert( ) discards all changes that have been made to thestorage object 426 since the last IStorage::Commit operation.EnumElements( ) returns an enumerator object that can be used toenumerate storage objects 426 and stream objects 424 contained within astorage object. DestroyElement( ) removes the specified storage object426 or stream object 424 from a storage object. RenameElement( ) renamesthe specified storage object 426 or stream object 424 in a storageobject. SetElementTimes( ) sets the modification, access, and creationtimes of the indicated storage element, if supported by the underlyingfile system. SetClass( ) assigns the specified CLSID to a storageobject. SetStateBits( ) stores state information in a storage object,for example up to 32 bits. Stat( ) returns the STATSTG structure for anopen storage object.

IStorageVB interface 350 is an automation friendly version of IStorageinterface 344. IStorageVB interface 350 supports the followingoperations, for example: Clear( ); Commit( ); Revert( ); sElementName(); bStorage( ); bElement( ); CreateStream( ); OpenStream( );Createstorage( ); OpenStorage( ); get_sName( ); getoStorage( );get_nCount( ); GetCompression( ); GetEncryption( ); GetCRC( );CreateStreamLinked( ); CreatePropertyStg( ); OpenPropertyStg( );SetClass( ); RegisterAlias( ); Destroy( ); and get_ElementType( ).Clear( ) clears the storage of all elements, e.g. sub-storages andstreams. Commit( ) ensures that any changes made to a storage objectopened in transacted mode are reflected in the parent storage. Fornon-root storage objects in direct mode, this method has no effect. Fora root storage, it reflects the changes in the actual device, forexample, a file on disk. For a root storage object open in direct mode,the commit( ) method is always called prior to releasing the object.Commit( ) flushes all memory buffers to the disk for a root storage indirect mode and will return an error code upon failure. Althoughreleasing the object also flushes memory buffers to disk, it has nocapacity to return any error codes upon failure. Therefore, callingreleasing without first calling commit( ) causes indeterminate results.Revert( ) discards all changes that have been made to the storage objectsince the last Commit( ) operation.

-   -   sElement( ) returns the name of the element. bStorage( ) returns        true if the element is a sub-storage. bElement( ) returns either        iStreamVB interface 412 or iStreamVB interface 414 or IStorageVB        interface 412 for the selected element. CreateStream( ) creates        and opens a stream object with the specified name contained in        the storage object. Nothing is returned if the stream cannot be        created. OpenStream( ) opens an existing stream object within        this storage object in the specified access mode. Nothing is        returned if the stream cannot be opened. CreateStorage( )        creates and opens a new storage object nested within the storage        object. Nothing is returned if the storage cannot be created.        OpenStorage( ) opens an existing storage object with a specified        name in the specified access mode. Nothing is returned if the        storage cannot be opened. Get_sName( ) returns the name of the        storage. Get_oStorage( ) returns the IDispatch interface, which        exposes objects, methods and properties to programming tools and        other applications that support Automation. COM components        implement the IDispatch interface to enable access by Automation        clients, such as Visual Basic.

Get_nCount( ) returns the count of elements in the storage.GetCompression( ) determines if streams may be compressed in the fileand if enabled streams may optionally be compressed when created.GetCRC( ) indicates whether a cyclic-redundancy-check (“CRC”), or adigital signature, check is to be performed on the file.CreateStreamLinked( ) creates a link to a stream in another POLESS file.CreatePropertyStg( ) creates a property storage. OpenPropertyStg( )opens a property storage. SetClass( ) assigns the specified CLSID to astorage object. RegisterAlias( ) registers an alias to a storage in thePOLESS file for access by the pluggable protocol. Destroy( ) destroysthe specified element. Get_ElementType( ) is a read-only command thatreturns the type of the element.

7) cPropertyStorage Class

cPropertyStorage class 412 implements IPropertyStorage interface 413,which supports the following operations, for example: ReadMultiple( );WriteMultiple( ); DeleteMultiple( ); ReadPropertyNames ( );WritePropertyNames( ); DeletePropertyNames( ); SetClass( ); Commit( );Revert( ); Enum( ); Stat( ); and SetTimes( ). ReadMultiple( ) readsproperty values in a property set. WriteMultiple( ) writes propertyvalues in a property set. DeleteMultiple( ) deletes property values in aproperty set. ReadPropertyNames( ) gets corresponding strung names frogiven property identifiers. WritePropertyNames( ) creates or changesstring names corresponding to given property identifiers.DeletePropertyNames( ) deletes string names for given propertyidentifiers. SetClass( ) assigns a CLSID to a property set. Commit( )flushes or commits changes to a property storage object, as is done withthe command IStorage::Commit, described previously. Revert( ) discardsall changes made since the last commit call when a property storage isopened in transacted mode. Enum( ) creates and gets a pointer to anenumerator for properties within a property set. Stat( ) receivesstatistics about a property set. SetTimes( ) sets modification,creation, and access times for a property set.

IPropertyStorageVB interface 414 is an automation friendly version ofIPropertyStorage interface 413 that manages the persistent properties ofa single property set. IPropertyStorageVB interface 414 supports thefollowing operations, for example: ReadVB( ); WriteVB( ); Delete( );CommitVB( ); RevertVB( ); SetClass( ); get_nCount( ); CopyTo( );GetName( ); WriteMultiple( ); and ReadMultiple( ). ReadVB( ) reads thevalue of a specified property from the property set. WriteVB( ) writes avalue for a specified property to the property set. If the property doesnot exist the property/value pair will be created. If the propertyalready exists, the value will be updated if opened in eAccess_Writemode. Delete( ) removes a property from the property set. CommitVB( )flushes or commits changes to a property storage object, as is done withthe command IStorage::Commit, described previously. RevertVB( ) discardsall changes made since the last commit call when a property storage isopened in transacted mode. SetClass( ) assigns the specified CLSID to aproperty storage object. Get_nCount( ) returns the count of propertiesin the property set. CopyTo( ) copies the contents of the sourceproperty set to a destination property set. GetName( ) returns the nameof the specified property. WriteMultiple( ) writes property values in aproperty set. ReadMultiple( ) reads property values in a property set.

8) cPropertyStorageAmalgamated Class

cPropertyStorageAmalgamated class 416 implementsIPropertyStorageAmalgamated interface 417, which supports the followingoperations, for example: PropertyStorageAdd( ) and ClearStorage ( ).PropertyStorageAdd ( ) adds a property set to the collection of propertysets. ClearStorage( ) clears the collection of property sets.

C. POLESS Exam Resource File

FIGS. 11 and 12-25 illustrate the POLESS layout of exam resource file120. Exam resource file 120 stores the various pieces of compiledinformation from exam source files 130, as shown in FIG. 5. Examresource file 120 contains all of the content required to deliver thetest. However, where the test is media-intense, exam resource file 120will contain the core elements for the test with “links” to the externalcontent. XXL compiler 140 and plugins 150 store the compiled informationto exam instance file 120 using one of IPersistResourceStream interface192 a, IPersistResourceSet interface 192 b, or IPersistResourceStoreinterface 192 to store the compiled information as a stream of data, aset of data, or a storage element, respectively. In a preferredembodiment, the layout of exam resource file 120 is in a hierarchicalPOLESS format that directly implements the format of the XXL testdefinition language. The test developer uses the XXL test definitionlanguage to create the logic files 230 and data files 212 (FIG. 5) ofexam source file 130. By having a storage structure that follows theformat of the XXL test definition language, the incremental accessaspect of POLESS is easily implemented. XXL compiler 140 determines thestorage location in exam resource file 120 that stores a particularpiece of compiled information, even information stored into examresource file 120 by one of plugins 150.

FIG. 11 illustrates the main storage branches of exam resource file 120,which corresponds to the top-level elements of the XXL test definitionlanguage, denoted by reference numeral 500. The main storage branches ofexam resource file 120 are, for example: exams branch 550; forms branch600; items branch 650; category branch 700; templates branch 750;sections branch 800; groups branch 850; plugins branch 900; data branch950; formGroups branch 1000; attributes branch 1050; scripts branch1100; and message box (“Msgbox”) branch 1150. Other storage branches mayalternatively be used.

Exam branch 550, as seen in FIG. 12, stores, for example, the primaryattributes, properties, and data that govern the test. Exam branch 550can store information for various tests, as is denoted by the three,vertical ellipses. A specific test is identified by the data stored inname attribute storage 552. Again, the various tests may each beidentified by a different name, as denoted by the solid border aroundname attribute storage 552 or other identification scheme. Attributesstorage 554 stores, for example, version information 555, and titleinformation 556 of the test as a stream of data or other data storageformat. Title information 556 is optional, as is denoted by the brokenborder. Any optional, customized information regarding the test isstored in custom properties 558 as a property storage or other datastorage format. Information relating to the forms of the test areoptionally stored in forms property storage 560. A form is a fixed orsubstantially fixed order of testing events. Many different forms can bestored in forms storage 560, giving flexibility to test driver 110 incontrolling progression of the test. FormGroups storage 562 optionallystores information relating to a collection of exam forms as a stream ofdata or other data storage format. Preferably, a single form from theformGroup is chosen to deliver to an examinee. The selection of the formfrom the group is performed by a selection plugin 160. Exam branch 550preferably contains at least one forms storage 560 either independentlyor within formGroups storage 562. Other information relating to the testmay be stored under exam branch 550. Other storage formats mayoptionally be used

Forms branch 600, as seen in FIGS. 13A and 13B, stores, for example, theprimary attributes, properties, and data that govern the progress of thetest. Forms branch 600 can store information for various forms, as isdenoted by the three, vertical ellipses. As described previously, a formis a fixed or substantially fixed or substantially fixed order oftesting events. A single form is identified by the data stored in nameattribute storage 602. Other identification formats may optionally beused. Again, the various forms may each be identified, for example, by adifferent name, as denoted by the solid border around name attributestorage 602. Attribute storage 604 stores, for example, begin sectioninformation 605, end section information 606, event information 607, andoptionally stores version information 608, title information 609, skipallowed information 610, restartable information 611, with information612, height information 613, and bit depth information 614. Allinformation stored in attribute storage 604 is stored as a stream ofdata or other data storage format. Begin section information 605 and endsection information 606 indicates, for example, respectively, whichsection of the test begins and ends the test.

Event information 607 indicates, for example, the order of events of thetest for that form. Each event has a name and is prefixed with an eventtype and a colon. Other formats are optional. The event type includes“section”, “report”, and “results”. Version information 608 and titleinformation 609 indicate the version and title of the form,respectively. Skip allowed information 610 indicates, for example,whether or not by default skipping of sections is allowed. Restartableinformation 611 indicates, for example, whether the form can berestarted. Any optional, customized information regarding the form isstored in custom storage 616 as a property set or other data storageformat. Timer storage 628 stores, for example, information relating tohow the form is to be timed as a storage element. Attributes storage 630stores, for example, the names of Timer Plugin 158 to be used with theform. Plugin data storage 632 and plugin data storage 633 store any datanecessary for timer plugin 158 as a storage element and a stream ofdata, respectively. Plugin data storage 632 and plug in data storage 633are optional. Scoring storage 634 stores, for example, informationrelating to the scoring of the form. Attributes storage 636 stores, forexample, the name of scoring plugin 164 to be used with the form. Plugindata 638 and plugin data 639 optionally store any data needed forscoring Plugin 164 as a storage element and a stream of datarespectively.

Items Branch 650, as seen in FIG. 14, stores, for example, the primaryattributes, properties, and data that govern the items, or testquestions, to be delivered to the examinee during the test. Items branch650 can store information for various items, as is denoted by the three,vertical ellipses. A single item is identified by the data stored inname attributes storage 652. Again, the various items may each beidentified by a different name, as denoted by the solid border aroundname attributes storage 652. Attributes storage 654 stores, for example,weight information 654, scored information 655, and optionally storesskip allowed information 656, title information 657, start information658, finish information 659, and condition information 660. Weightinformation 654 indicates, for example, a value used for judging andscoring the item. In one embodiment, by default an item is given aweight of one in accordance with one embodiment, but other values may beutilized. Scored information 655 indicates, for example, whether or notthe item is scored as opposed to whether the item is being used as anexample. The default of scored information 655 is true. Skip allowedinformation 656 indicates, for example, whether the examinee can skipthe item without answering.

Start information 658 indicates, for example, script execution at thebeginning of the item and finish information 659 indicates, for example,script execution at the end of the item. Condition information 660indicates, for example, whether or not there is a condition on the itembeing delivered to the examinee. The information stored in attributesstorage 654 is stored as a stream of data or other data storage format.Data storage 662 and data stream 664 store any information regarding theproperties of the item. For example, data storage 662 or data stream 664can store the correct answer of a multiple choice item. Data storage 662and data stream 664 stored the information as a storage element and astream of data respectively.

Any optional, customized information regarding the item is stored incustoms storage 666 as a stream of data or other data storage format.Category storage 668 stores, for example, information relating to eachcategory to which the item belongs. The information stored in categorystorage 668 preferably and optionally is redundant, as category branch700 stores, for example, all the items within the specific categories.The reason for the optional redundancy is so that test driver 110 canquickly look up the category of any item.

Category branch 700, as seen in FIG. 15, stores, for example, theprimary attributes, properties, and data that govern the testcategories. A test category provides a grouping mechanism, which isindependent of delivery of the test, allowing for exotic reporting andscoring if necessary. Category branch 700 is optional as denoted by thebroken border. Category branch 700 can store information for variouscategories, as is denoted by the three, vertical ellipses. A singlecategory is identified by the data stored in name attributes storage702. Again, the various categories may each be identified by a differentname, as denoted by the solid border around name attributes storage 702.Attributes storage 704 stores, for example, complete information 705,duplicates information 706, contents information 707, and optionallystores, for example, description information 708. Complete information705 indicates, for example, whether or not every item in the categorymust appear within the category or within its subcategories. Duplicatesinformation 706 indicates, for example, whether the item can appear morethan once within the category or within the subcategories. Contentsinformation 707 determines what can exist within a category.

Description information 708 is used within the category to contain adescription of the category's contents. Category storage 710 stores, forexample, information relating to any subcategories under the categoryidentified in name attribute storage 702. Items storage 712 indicates,for example, any items that exist within the category. Sections storage714 contains information indicating what any sections that exist withinthe category. Scoring storage 716 contains information relating to thescoring of the items within the category. Attributes storage 718 stores,for example, the name of the scoring plugin to be used with the item.Data storage 720 and data stream 722 contain the information needed toinitialize scoring plugin 164. Data storage 720 and data stream 722store the information as a storage element and a stream of datarespectively.

Templates branch 750, as seen in FIG. 16, stores, for example, theprimary attributes, properties, and data that govern the templates usedin the test. Template branch 750 can store information for various maintemplates, as is denoted by the three, vertical ellipses. A single maintemplate is identified by the data stored in name attributes storage752. Again, the various templates may each be identified by a differentname, as denoted by the solid border around name attributes storage 752.Attributes storage 754 stores, for example, split information 756, orderinformation 757, and optionally stores size information 759. Splitinformation 656 defines how a specific area within the template is to besplit or separated, for example, either by rows or columns or othershapes and/or sizes. Size information 759 indicates, for example,possible values for describing the size of the template, for example,pixels, percentages, or html syntax. Template storage 760 stores, forexample, information relating to any sub-templates to be used under thetemplates specified by the information in name attributes storage 752.Sub-templates are identified by the information in name attributesstorage 762. Many sub-templates 760 can exist as denoted by the threevertical ellipses.

Areas storage 764 indicates, for example, information relating to theareas used within the template denoted by the information in nameattributes storage 752. Many areas may exist within a template asdenoted by the three vertical ellipses. Each area is identified by theinformation stored in name attribute storage 766. Attribute storage 768stores, for example, visible plugin name information 760, sizeinformation 770, and allow more information 771. Plugin name information760 indicates, for example, the name of the visible plugin to be usedwith the area. Size information 770 indicates, for example, the size ofthe area, as for example a pixel value, a percentage value, or HTMLsyntax. Plugin data 772 and plugin data 774 store information relatingto the visible plugin to be used in the area. The data stored in eitherplugin data storage 772 or plugin data stream 774 is executed by thevisible plugin when the template is loaded. Plugin data storage 772 andplugin data stream 774 stores, for example, the information as either astorage element or a stream of data, respectively. Other information mayoptionally be stored.

Section branch 800, as seen in FIG. 17, stores, for example, the primaryattributes, properties, and data that govern test sections. Testsections dictate the navigation and timing of groups of items as well asdisplays within the test. Sections branch 800 can store information forvarious sections, as is denoted by the three, vertical ellipses. Asingle section is identified by the data stored in name attributestorage 802. Again, the various sections may each be identified by adifferent name, as noted by the solid border around name attributesstorage 802. Attributes storage 804 stores, for example, groupinformation 805 and optionally stores title information 806, skipallowed information 807, start information 808, finish information 809,and condition information 810. Group information 805 indicates, forexample, to which group of the test the section belongs. Skip allowedinformation 807 indicates, for example, whether or not the items withinthe section may be skipped. Start information 808 indicates, forexample, script execution at the beginning of the section and finishinformation 809 indicates, for example, script execution at the end ofthe section. Condition information 810 indicates, for example, anyconditions that exist regarding the section. Any optional, customizedinformation regarding this section is stored in custom property storage812 as a stream of data or other data storage format. Custom attributeswill be stored as a property set. The “key” for each attribute will be astring or other acceptable format.

Timer storage 814 stores information regarding, for example, the timingof the section. Attribute storage 816 stores, for example, informationidentifying timer plugin 158, which is to be used with a section. Plugindata storage 818 and plugin data storage 820 stores, for example, dataneeded for timer plugin 158. Plugin data storage 818 and plugin datastorage 820 stores, for example, information as a storage element and astring of data, or other acceptable format, respectively. Navigationstorage 822 stores, for example, information relating to the delivery ofpresentations and groups within the section. Attributes storage 824stores, for example, information indicating which navigation plugin 162is to be used with this section. Plugin data storage 826 and plugin datastream 828 store information needed for the navigation plugin 162.Plugin data storage 826 and plugin data stream 828 store the informationas a storage element and a stream of data respectively. Groups branch850, as seen in FIG. 18, stores, for example, the primary attributes,properties, and data that govern the groups within the test. A groupdetermines the order of events within the test. Groups branch 850 canstore information for various groups, as is denoted by the three,vertical ellipses. A single group is identified by the data store inname attributes storage 852. The various groups may each be identifiedby a different name, as noted by the solid border around name attributesstorage 852. Attributes storage 854 stores, for example, typeinformation 855, event information 856, title information 857, andreviewed name information 858. Type information 855 indicates, forexample, whether the group is either a “group holder” (group ofpresentations), or a “section holder” (group of sub-sections). These aremutually exclusive.

Event information 856 indicates, for example, the order of events withinthe test. Review name information 858 indicates, for example, whether ornot a presentation within the group is to be used as a review screen.Any optional, customized information regarding the group is stored incustom storage 860 as a stream of data or other data storage format.Events storage 862 stores event information, for example, as isdescribed in further detail in FIG. 19. Scoring storage 864 stores, forexample, information relating to the scoring of items within the group.Attributes storage 866 stores, for example, information indicating whichscoring plugin 164 is to be used with the group. Selection storage 872stores, for example, information relating to the selection of itemswithin the group. Attributes storage 874 indicates, for example, whichselection plugin 160 is to be used with the group.

FIGS. 19A, 19B, 19C, and 19D illustrate the events sub-branch of groupsbranch 850 in greater detail, in accordance with one embodiment of theinvention. In FIG. 19A, events sub-branch 862 can store information forvarious events. For example, events sub-branch 862 is storinginformation in events name sub-branch 880, event name sub-branch 890,and event name sub-branch 897. Attributes storage 881, in FIG. 19B,under events name storage 880 stores, for example, type information 882,template information 883, and optionally stores title information 884,counted information 885, start information 886, finish information 887,and condition information 888. Type information 882 indicates, forexample, whether the event is an item or a display. Template information883 indicates, for example, which template is being used with the event.Counted information 885 indicates, for example, whether a presentationshould be included in the totals of presentations presented to theexaminee in a section. Generally, presentations with items, orquestions, are counted and introductory presentations are not counted.

Start information 886, finish information 887, and condition information888 indicates, for example, start, finish, and conditional scriptsrespectively. Any optional, customized information regarding the eventis stored in custom storage 889. The “key” for each custom attributewill be a string. Referring again to FIG. 19A, event name storage 890indicates, for example, a different event, which contains differentattributes. Additionally, area information 891, in FIG. 19B, indicates,for example, which area is rendering the presentations content and iteminformation 892 indicates, for example, the name of the associated itemif the event is of the item type. Additionally, data storage 893, datastream 894, data storage 895, and data storage 896 contain informationused in a nested presentation. The data off of a nested presentation arethe contents of the item or the presentation. This data may be a stream,a storage, a link to a stream, a link to a storage, or other format. InFIG. 19C, event name 897 indicates, for example, another event, whichincludes a sub-event 898, in FIG. 19D.

Plugins branch 900, as seen in FIG. 20, stores, for example, the primaryattributes, properties, and data that govern any plugins 150 used forthe test. Plugins branch 900 can store information for various plugins,as is denoted by the three, vertical ellipses. A single plugin isidentified by the data stored in name attribute storage 902. A CLSID isstamped with the name of the plugin 150. Attributes storage 904 stores,for example, information identifying the plugin 150 by a program ID.Data storage 906 and data storage 908 store initial data for the pluginas either a storage element or a stream of data respectively.

Data branch 950, as indicated in FIG. 21, stores, for example, anyglobal data needed for the test. Data stored optionally under databranch 950 may be stored as either a storage element or a stream of dataas indicated by data storage 952 and data storage 954. Data stored underdata branch 950 may be directly used by a plugin 150 or the data may beresources (.gif, .jpeg, .wab, .mpeg, etc.) used internally by a plugin150.

FormGroups branch 1000, as seen in FIG. 22, stores, for example, theprimary attributes properties and data that govern the formGroups of thetest. FormGroups branch 1000 can store information for variousformGroups, as is denoted by the three, vertical ellipses. A singleformGroup is identified by the data stored in name attributes storage1002. The various formGroups may each be identified by a different name,as denoted by the solid border around name attributes storage 1002.Attributes storage 1004 stores, for example, information indicatingwhich forms are to be used within the formGroup. Selections storage 1006stores, for example, information relating to the selection of itemswithin the formGroup. Attributes storage 1008 indicates, for example,which selection plugin 160 is to be used with the formGroup. Plugin datastorage 1010 and plugin data storage 1012 store any information neededfor the selection plugin 160. Attributes storage branch 1050 stores, forexample, attribute information that is global to exam resource file 120.This includes the last execution state of XXL compiler 140 [sMode], themajor [iXXLMajorVersion] and the minor version [iXXLMinorVersion] of theXXL language.

Scripts branch 1100 stores, for example, information relating to scriptsused within the test. Attributes storage 1102 stores, for example, typeinformation that specifies which type of language the script is in, forexample, VB script of J script. Scripts storage 1104 stores, forexample, global scripts used within the test that may be referenced bythe test driver. MsgBox branch 1150 stores, for example, informationrelating to the size and content of any message boxes that may bedelivered to the examinee during the test. Message boxes may betriggered by plugins 150 during the exam.

D. POLESS Exam Instance File

FIGS. 26A, 26B, 26C, and 26D illustrate the POLESS layout of examinstance file 170. Exam instance file 170 stores information regardingthe current examinee's test. Exam instance file 170 is created when atest starts for an examinee. Exam instance file 170 is destroyed whenthe test successfully completes. If the examinee must restart her testdue to some interruption, for example, a power failure, the state of thetest is restored from Exam instance file 170. In a preferred embodiment,the layout of exam instance file 170 is in a hierarchical POLESS format.As seen in FIG. 26A, the top-level storage branches of exam instancefile 170 from root 1200 are, for example: running branch 1202; contentsbranch 1310; and history branch 1320. Root 1200 relates to POLESScStorageRoot class 406 (FIG. 10), which instantiates exam instance file170.

Running branch 1202 stores, for example, the state information of allrunning objects in test driver 110 and plugins 150. Plugins 150 use oneof IPersistInstanceStream interface 196 a, IPersistInstanceSet interface196 b, or IPersistInstanceStore interface 196 c to store information toexam instance file 170 as a stream of data, a set of data, or a store ofdata, respectively. Any of plugins 150, except display plugin 152,results plugin 166, report plugin 168, and helm plugin 154, which do notcontain examination state information, store examination stateinformation to exam instance file 170. Test driver 110 determines thestorage location in exam instance file 170 that stores a particularpiece of examination state information.

Exam sub-branch 1204 contains examination state information relating tothe exam. Contents storage 1206 stores, for example, exam statusinformation 1207 and version information 1208. Exam status information1207 indicates, for example, the status of the exam, for example,initializing or terminating. Template storage branch 1210 stores, forexample, examination state information relating to templates running inthe exam. Name attribute storage 1212 stores, for example, countinformation 1214 and observed ever information 1215. Observed everinformation 1215 indicates, for example, whether or not the template'scontent has ever been fully seen by the examinee.

Form storage branch 1216 contains information relating to the forms usedwithin the exam. Contents storage branch 1218 stores, for example,seconds information 1219, date start information 1220, date finishinformation 1221, current section information 1222, and versioninformation 1223. Current section information 1222 indicates, forexample, the current section being delivered to the examinee in theform. Version information 1223 indicates, for example, theidentification of the form.

Sections chosen storage branch 1224, as illustrated in FIG. 26B, stores,for example, information relating to sections in the form beingdelivered to the examinee. Contents storage 1226 stores, for example,the names of the sections that have been or will be delivered to theexaminee. Name attribute storage 1228 indicates, for example, the nameof a particular section. Contents storage 1230 stores, for example,current child information 1231, seconds information 1232, date startinformation 1233, and date finish information 1234. Navigation storage1236 and navigation storage 1237 store the state information ofnavigation plugin 162. Navigation storage 1236 stores, for example, theexamination state information from navigation plugin 162 if navigationplugin 162 implements the IPersistInterfaceSet 196 b orIPersistInterfaceStore 196 c. Navigation storage 1237 stores, forexample, the information from navigation plugin 162 if navigation plugin162 implements IPersistInterfaceStream 196 a. Timers storage 1238 andtimers storage 1239 store information from timer plugin 158. Timerstorage 1238 is used if timer plugin 158 implements IPersistInterfaceSet196 b or IPersistInterfaceStore 196 c. Timers storage 1239 is used iftimer plugin 158 uses IPersistInterfaceStream 196 a.

Items chosen sub-branch storage 1240 stores, for example, informationrelating to items that have been or will be delivered to the examinee.Contents storage branch 1242 stores, for example, the names and order ofall the items that have been or will be delivered to the examinee. Nameattributes storage 1244 indicates, for example, the identification of aparticular item. Contents storage branch 1246 stores, for example,presented information 1244, complete information 1248, skippedinformation 1249, seconds information 1250, dehydrated information 1251,and observed ever information 1252. Presented information 1247indicates, for example, whether the item has ever been delivered to theexaminee. Completed information 1248 indicates, for example, whether ornot the item has been completed. Skipped information 1249 indicates, forexample, whether the item has been skipped. Item plugin storage 1254 anditem plugin storage 1255 stores, for example, examination stateinformation from item plugin 156. Item plugin storage 1254 is used ifitem plugin 156 uses IPersistInterfaceSet 196 b orIPersistInterfaceStore 196 c. Item plugin storage 1255 is used if itemplugin 156 uses IPersistInterfaceStream 196 a.

In FIG. 26C, item light storage 1256 exists only if the item wasdehydrated (to save memory or when a section ends). The dehydrated itemstores the data but actions on the data are no longer available untilthe item is re-hydrated. Item light storage 1256 stores, for example,score candidate information 1257. Score minimum information 1258, scorenominal information 1259, score maximum information 1260, completeinformation 1261, skipped information 1262, correct answer display 1263,response results 1264, and correct answer results 1266. Timers storage1268 and timers storage 1269 store information from timer plugin 158.Timer storage 1268, as seen in FIG. 26B, is used if timer plugin 158implements IPersistInterfaceSet 196 b or IPersistInterfaceStore 196 c.Timers storage 1269 is used if timer plugin 158 usesIPersistInterfaceStream 196 a. Score storage 1270 and Score storage 1271store information from timer plugin 158. Timer storage 1270 is used iftimer plugin 158 implements IPersistInterfaceSet 196 b orIPersistInterfaceStore 196 c. Score storage 1271 is used if timer plugin158 uses IPersistInterfaceStream 196 a.

In FIG. 26C, groups chosen sub-branch storage 1272 indicates, forexample, which groups have been or will be delivered to the examinee.Contents storage 1274 stores, for example, the names of the groups. Nameattributes storage 1276 indicates, for example, the name of a particulargroup. Contents storage 1278 stores, for example, names of groups andthe order of groups. Scoring storage 1280 and scoring storage 1281 storeexamination state information from score plugin 164. Scoring storage1280 is used if score plugin 164 implements IPersistInterfaceSet 196 bor IPersistInterfaceStore 196 c. Scoring storage information 1281 isused if score plugin 164 implements IPersistInterfaceStream 196 a.Selection storage 1282 and selection storage 1283 store information fromselection plugin 160. Selection storage 1282 is used if selection plugin160 implements IPersistInterfaceSet 196 b or IPersistInterfaceStore 196c. Selection storage 1283 is used if selection plugin 160 implementsIPersistInterfaceStream 196 a. Delivered storage 1284, in FIG. 26D,stores, for example, an ordered list of groups chosen for delivery.Delivered storage 1285 stores, for example, an ordered list of thesub-classes of the form, for example: sections, reports and results.

Presentations chosen storage sub-branch 1286 indicates, for example, anypresentations that have been or will be delivered to the examinee.Contents storage 1288 stores, for example, the names of thepresentations. Names storage sub-branch 1290 stores, for example, thename of the presentation. Names storage 1290 also stores, for example,comment information 1291, marked information 1292, count information1293, name information 1294, observed ever information 1295, nameinformation 1296, and observed ever information 1297. Name information1294 and observed information 1295 relate to the name of the firstpresentation area stored under presentations chosen sub-branch 1286 andwhether or not the presentation has ever been observed, and nameinformation 1296 indicates, for example, the last presentation area thatwas delivered to the examinee and whether or not the presentation wasever observed. Contents storage 1298 stores, for example, informationleading to events. Contents storage 1298 stores, for example, readyinformation 1299 ever checked information 1300, ever started information1301, and ever finished information 1302. Ready information 1299indicates, for example, whether the event is ready to be delivered tothe examinee. Ever checked information 1300 indicates, for example,whether an event's conditional delivery script ever been checked.Preferably, the conditional delivery script is only checked once. Everstarted information 1301 indicates, for example, whether the event wasever started by the examinee. Ever finished information 1302 indicates,for example, whether the event was completed by the examinee.

Referring again to FIG. 26A, contents branch 1310 stores, for example, aproperty set containing information to identify the examination instanceand the examination start count 1312. The identifying information usedis the examinee appointment identification 1311, the name 1313 of examresource file 120, and the name 1314 of the specified form or group.

History branch 1320 is a single stream of chronological text messagesthat logs the history of the test. These text messages are used by staffat system headquarters to diagnose problems that occurred in the field.Each text message is prefixed with the date, time, and a level ofseverity, for example: information, warning, or error. Test driver 110will filter the text messages to a level of diagnostics desired for testdriver 110, such as determining errors in test driver 110 or detailhistory tracking, including general information.

V. Expansion of Test Driver Using Plugins

FIG. 27 illustrates the process for customizing test based on specificrequirement from the client using plugins 150, denoted generally byreference numeral 1400. First, the client presents the new requirements,for example, a new item type, to the test developer, step 1402. The testdeveloper then writes and XML schema to define the XXL testspecification, step 1404. The schema is subsequently used to validatethe XXL test specification.

A detailed description of the XXL schema is given in U.S. PatentPublication No. 20030129573, entitled “EXTENSIBLE EXAM LANGUAGE (XXL)PROTOCOL FOR COMPUTER BASED TESTING,” incorporated herein by reference.

The test developer next writes the appropriate plugin 150, in thisexample, item plugin 156. The test developer also implements the IPlugininterface 167 and IPlugin interface and IItem interfaces 169.Additionally, the test developer implements IPersistResource interface192 (FIG. 3) to enable persistence of compiled test information fromitem plugin 156 to exam resource file 120. The test developer canoptionally implement IPersistInstance interface 196 (FIG. 3), step 1408,to enable persistence of examination state information from item plugin156 to exam instance file 170. After the appropriate interfaces havebeen implemented, item plugin 156 is valid and operating. Finally, afterthe test is delivered to the examinee, the result processor accumulatesresults from the examinee, 1410. The results processor must be able tounderstand the new item type to correctly process the results.Customization process 1400 only required the test developer to write onepiece of software, item plugin 156, to accommodate the client'scustomizations rather than multiple pieces of software.

A. Test Production and Test Delivery

FIG. 28 is a flow chart illustrating the overall method of testproduction and test delivery, denoted generally by reference numeral1500. The test publisher first authors the test specification andcontent, step 1502. The test specification and content are then storedin exam source files 130, step 1504. Exam source files 130, for examplethe content of XXL files 134, are then compiled and validated, step1506. The compiled test specification and content are stored in examresource file 120, step 1508. Finally, the compiled test specificationand content are delivered to the examinee, step 1510.

The validation of the test specification and content is illustrated ingreater detail in FIG. 29, by the method denoted generally by referencenumeral 1512. When the test specification and content stored in examsource files 130 specifically references a plugin 150, that plugin 150is instantiated, step 1514. Partial test specification and contentrelating to that plugin 150 are loaded into the plugin 150 from examsource files 130, step 1516. In an alternative embodiment, the partialtest specification and content are loaded into a private memory in datacommunication with the plugin 150. The plugin 150 validates the partialtest specification and content, step 1518. The validated testspecification and content are then unloaded from the plugin 150 into astorage element within exam resource file 120.

FIG. 30 illustrates the method of the test delivery cycle in greaterdetail. When the previously validated test specification and contentstored in exam resource file 120 references a plugin 150, the plugin 150is instantiated, step 1525. The storage element in exam resource file120 containing the validated test specification and content are providedto the plugin 150, step 1527. The validated test specification andcontent are loaded into the plugin 150 from the storage element withinexam resource file 120, step 1529. Finally, the examination stateinformation, which includes, for example, the examinee's responses, isstored into exam instance file 170, step 1533.

FIG. 31 illustrates the method of restarting a test after interruptionin greater detail. In test restart method 1535, test driver 110 isstarted, step 1537. Test driver 110 determines whether the test hasalready started, step 1539. If the test delivery has not alreadystarted, plugins 150 reload validated test specification and contentfrom exam resource file 120, step 1543. If the test has already started,plugins retrieve examination information from exam instance file 120,step 1541. Plugins 150 then reload the validated test specification andcontent from exam resource file 120, step 1543. Test driver 110 thendelivers the exam to the examinee, step 1545.

B. Plugin Life Cycle

FIG. 32 illustrates the life cycle of plugin 150 from test production totest delivery, denoted generally by reference numeral 1420. Dashedvertical line 1422 divides the plugin life cycle 1420 into a testproduction cycle, to the left of dashed vertical line 1422, and a testdelivery cycle, to the right of dashed vertical line 1422. The testproduction cycle occurs only occasionally when new plugins 150 aredeveloped to satisfy the requirements of a client. The test deliverycycle occurs whenever the test is delivered to the examinee, forexample, daily.

Exam source files 130, of which data files 132 and XXL files 134 areshown, contain every aspect of the test as written by the testpublisher. In step I, XXL compiler 140 reads from XXL files 134 andinterprets instructions that call for the use of a plugin 150. Plugin150 is identified in the XXL test definition language by both a name anda program identification (“prog ID”). When XXL compiler receives theprog ID from XXL files 134, XXL compiler knows that a plugin 150 isrequired to complete the compilation of exam source files 130.

Not all of the possible types of plugins 150 are required to build anyone test. Also, more than one plugin 150 is implemented for a specifictype. In the above example, two navigation plugins 162 and two itemplugins 156 are defined. XXL compiler 140 reads information from examsource files 130 using IStream interface 340, iNode interface 1424,which is the Microsoft interface used to access a node of an XMLdocument in the document object model (“DOM”), and IStreamVB interface348. XXL compiler 140 instantiates the requested plugin 150 using, forexample, the call CoCreateInstance( ). CoCreateInstance( ) creates asingle, uninitialized object of the class associated with a specifiedCLSID, using a prog ID that has been converted into the CLSID.

If the data referring to plugin 150 has been customized by the testdeveloper, XXL compiler 140 may not recognize the new data. Therefore,XXL compiler 140 passes the data directly to plugin 150 and plugin 150loads the data into a private memory (not shown). In one embodiment, theprivate memory is internal to plugin 150, and in another embodiment, theprivate memory is external to plugin 150. Plugin 150 can then validatethe data using the XXL schema. If the data is invalid, plugin 150reports the error. In an alternative embodiment, plugin 150 can validatethe data using an XML document type definition (“DTD”). A DTD is aformal description in XML Declaration Syntax of a particular type ofdocument. Similar to a schema, a DTD sets out what names are to be usedto the different types of elements, where they may occur, and how theyall fit together. However, the XXL schema is preferred for validationsince schemas are easier to read than a DTD and are very flexible.

If plugin 150 declares that the data is valid, XXL compiler 140 preparesa POLESS storage object 300 in exam resource file 120 to which plugin150 saves the data at a command from XXL compiler 140, in step II. Asdescribed previously, XXL compiler 140 determines where the data fromplugin 150 is to be saved in exam resource file 120 and creates theappropriate storage location. The name, CLSID, and data associated withplugin 150 is stored in plugins branch 900 in exam resource file 120(FIG. 20). Plugin 150 implements IPersistResource interface 192 to storethe data to exam resource file 120. Data storage 906 stores, forexample, the data, for example, as either a stream, set of data, or as astorage element if plugin 150 implements either IPersistResourceStream192 a, IPersistResourceSet interface 192 b, or IPersistResourceStoreinterface 192 c, respectively. Data storage 908 stores, for example, thedata as a stream of data if plugin 150 implements IPersistResourceStreaminterface 192 a. Plugin 150 can choose the format used to store the datainto exam resource file 120. Steps I and II are repeated until examsource files 130 are completely compiled and exam resource file 120 iscompletely populated with the compiled test information.

The compile sequence of a plugin 150, as shown in steps I and II in FIG.32, are illustrated in greater detail in FIG. 33. Plugin compilesequence 1430 begins as XXL compiler 140 asks plugin 150 to validate theinformation from exam source files 130 that pertain to plugin 150 usingIPlugin::ValidateSource( ) call 1432, in step I. Plugin 150 validateswhether or not the data received from exam source files 140 is correctlyformatted based on the XXL schema. If the data is not valid, pluginthrows a structured COM error. Plugin 150 does not validate that allrequired source elements are present, but rather, that what is presentis correctly formatted.

Step 1I contains two steps, indicated as step IIa and IIb. In step IIa,XXL compiler 140 creates the appropriate storage element in examresource file 120 using POLESS object 300. The storage element type isdetermined based on the type of IPersistResource interface 192 thatplugin 150 implements, for example: IPersistResourceStream interface 192a; IPersistResourceSet interface 192 b; or IPersistResourceStoreinterface 192 c. XXL compiler 140 then calls IPersistResource*::Save( )call 1434 for the appropriate IPersistResource interface. Plugin 150saves the compiled information from exam source files 130 to examresource file 120 through the POLESS object 300 passed by XXL compiler140. In step IIb, XXL compiler 140 instructs plugin 150 to unload, orflush, its content using Unload( ) call 1436. As stated previously,steps I, IIa, and IIb are repeated until all of exam source files 130 iscompiled.

Step VI, which is shown as steps VIa and VIb, concerns amalgamation ofexam resource file 120. Amalgamation enables data for a specific pluginto exist virtually as one storage location even if the data appears atdifferent locations within the storage hierarchy. Amalgamation can beperformed on exam resource file 120 if plugin 120 has implemented eitherIPersistResourceSet interface 192 b or IPersistResourceStore interface192 c which storing data to exam resource file 120. In step VIa, XXLcompiler 140 amalgamates one to three storage elements in exam resourcefile 120 and passes the amalgamated POLESS object to plugin 150 usingIPersistResource*::ValidateResource( ) call 1438. Plugin 150 determineswhether or not the amalgamated POLESS object creates a complete andvalid set. Plugin 150 throws a structured COM error if the amalgamatedPOLESS object does not create a complete and valid set. In step VIb, XXLcompiler 140 instructs plugin 150 to unload, or flush, its content usingUnload( ) call 1440. Steps VIa and VIb are interspersed among steps I,IIa, and IIb cycles and can also occur multiple times during thecompilation of exam source files 130. Amalgamation is described ingreater detail, in U.S. Patent Publication No. 20030129573, entitled“EXTENSIBLE EXAM LANGUAGE (XXL) PROTOCOL FOR COMPUTER BASED TESTING,”incorporated herein by reference.

Referring again to FIG. 32, during the test delivery cycle, test driver110 reads the test specifications stored in exam resource file 120through POLESS objects 300. Test driver 110 reads information from examresource file 120 through POLESS objects 300 in order to retrieve theencrypted, compressed, and structured elements within exam resource file120. When the XXL test definition language calls a plugin 150 by a progID, as described previously, test driver 110 instantiates the plugin 150that was called, in step III. Test driver 110 provides the POLESS object300 from exam resource file 120 and plugin 150 initializes itself fromthe POLESS object 300, for example, data storage 906 or data storage 908stored under name attribute storage 902, using the appropriateIPersistResource interface 192. The information loaded into plugin 150is the same information as was stored into exam resource file 120 byplugin 150 during the test production cycle (step II). Since plugin 150chose the storage format used to store the information into examresource file 150, plugin 150 can always read the information from examresource file 150, giving plugin 150 complete flexibility. Test driver110 need not be able to read the information that is used by plugin 150.Therefore, any customizations to the test facilitated by plugin 150 doesnot require any changes to test driver 110. The test then progresseswith plugin 150 enhancing the functionality of test driver 110 based onthe new requirements from the client.

Periodically, based on a request either from test driver 110 or fromplugin 150, the state of all running objects will save to exam instancefile 170, which is a unique file for each examinee, indicating theprogress and the status of the test for that examinee. Test driver 110asks plugin 150 if plugin 150 is “dirty,” meaning that plugin 150 isstoring has some updated examination state information. For example,when the examinee selects distractor A on a multi-choice item, itemplugin 156, in this case, becomes dirty. If plugin 150 is dirty, testdriver 110 provides plugin 150 a POLESS object 300 in exam instance file170 and plugin saves the examination state information to exam instancefile 170 using IPersistInstance interface 196, in step IV. For example,item plugin 156 saves the examinee's answer to item plugin storage 1254or to item plugin storage 1255 (FIG. 26). Item storage 1254 stores, forexample, the data as either a set of data or as a storage element ifitem plugin 156 implements either IPersistInstanceSet interface 196 b orIPersistInstanceStore interface 196 c, respectively. Item storage 1255stores, for example, the data as a stream of data if item plugin 156implements IPersistInstanceStream interface 196 a.

Step V occurs if the test is interrupted, for example, because of apower failure, and the test needs to restart. When test driver 110 isrequired to return to a particular operation state, test driver 110reads the examination state information from exam instance file 170.Plugin 150 is provided the storage object containing the state of plugin150 as saved in step IV using IPersistInstance interface 196. Using theprevious example, item plugin 156 retrieves its state information fromitem plugin storage 1254 or for item plugin storage 1255. Plugin 150 isable to become operational from the retrieved state information,enabling a restart of the test from the point at which the test wasinterrupted.

The delivery sequence of a plugin 150, as shown in steps II, IV, and Vin FIG. 32, are illustrated in greater detail in FIGS. 34A, 34B, 34C,and 34D. As seen in FIG. 34A, delivery sequence 1520 particularlyrelates to visible plugins 150, e.g., display plugin 152, helm plugin154, and item plugin 156. Step III contains sub-steps labeled IIIa-IIIb.Plugin delivery sequence 1520 begins, in step 111 a, when the currentdelivering presentation requests its template to activate withcTemplate::Activate( ) call 1524. Activate( ) call 1524 is activatedwhen the examinee navigates on the test using a helm navigation controlactivated by helm plugin 154. IContainerNotifyHelm interface 206 allowshelm plugin 154 to request navigation from test driver 110.IContainerNotifyHelm interface 206 sends Activate( ) call 1524 tocTemplate class 236 in test driver 110 (see FIG. 8).

In step 111 b, cTemplate class 236 in test driver 110 usesIPlugin::Load( ) call 1526 to set the core object references from testdriver 110 into the plugin 150 being delivered. The core objectreferences include IContainerNotify interface 200, the cExam class (notshown), and the IAppointment interface 176, which passes informationregarding the examinee and appointment to plugin 150.

Step V, which is interspersed with step III, occurs only if the test isinterrupted and plugin 150 loses state. cTemplate class 236 in testdriver 110 uses IPersistInstance*::Reload( ) call 1528 to call on thereload method of exam instance file 170. Exam instance file 170 reloadsplugin 150, through IPersistInstance interface 192, for example,IPersistInstanceSet 192 b, with the state saved to the appropriatestorage location in exam resource file 170 (see FIG. 26).

Step IIIc is performed for both initial delivery of plugin 150 andduring restart of the test, in conjunction with step V. cTemplate class236 in test driver 110 uses IPersistResource*::Load( ) call 1530 to callon the load method of exam resource file 120. Exam resource file 120loads plugin 150, through IPersistResource interface 192, for exampleIPersistResourceSet interface 192 b, with the test specification andcontent from the appropriate storage location in exam resource file 120.Plugin 150 is loaded with test specification and content from examresource file 120 when being initially delivered to the examinee. Plugin150 is also loaded with test specification and content from examresource file 120 and with examination state information from examinstance file 170, as described above, when the test has beeninterrupted and plugin 150 must recover state.

After plugin 150 is properly loaded, cTemplate class 236 in test driver110 uses, I*::PresentationStarting( ) call 1532 (continued in FIG. 34B)to inform visible plugin 150 that the presentation is starting, in step111 d. I*::PresentationStarting( ) call 1532 is made to any visibleplugin 150 being used in the presentation on the appropriate interface,for example: IDisplay interface 169 a, IItem interface 169 c, or IHelminterface 169 b. For example, an IItem::PresentationStarting( ) call isused for item plugin 156. cTemplate class 236 then instruct visibleplugins 150 to display using IOleObject::DoVerb(Show, . . . ) command1534, step IIIe. IOleObject interface 1522 is the Active Documentinterface used to implement the Active Document presentation. IOleObjectinterface 1522 is the combination of the Active Document interfacesdescribed in conjunction with FIG. 7. After instructing visible plugins150 to display, test driver 110 awaits notification from each visibleplugin 150 that the specific visible plugin 150 has successfully shown.Visible plugins 150 call back to test driver 110 usingIContainerNotify::Activated( ) call 1536, step IIIf (continued in FIG.34B). Now, the presentation is started and active such that the examineecan interact with the presentation.

The deactivation of the presentation begins with a request from the helmfor navigation. For example, if the examinee has finished a question andwishes to move on to the next question on the next presentation, theexaminee can choose the “NEXT” button on the helm. The navigationrequest is sent from IHelm interface 169 b, which receives the requestfrom the examinee, to test driver 110 using IContainerNotifyHelminterface 206. As seen in FIG. 34D, the request is made usingIContainerNotifyHelm::Request-Move( ) call 1538, step IIIg. Test driver110 then asks each item plugin 156 being used in the presentationtemplate if the examinee is allowed to leave the current presentationand to proceed to the next presentation. The query is made usingIItem::bProceed( ) call 1540, step IIIh. If all item plugins 156 respondin the affirmative, test driver 150 passes the navigation request tonavigation plugin 162, which is an invisible plugin 150. Test driver 110passes the request using INavigate::RequestMove( ) call 1542, step IIIl.Navigation plugin 162 determines the resultant location of the requestednavigation. In FIG. 34, for example, navigation plugin 162 determinesthe section of the test to which the examinee will proceed usingISection::ChildNext( ) call 1544, step IIIj.

The active presentation then instructs the template to deactivate usingcTemplate::Deactivate( ) call 1546, step IIIk (continued in FIG. 34C).Referring back to FIG. 34D, template class 236 in test driver 110requests that visible plugins 150 hide from the Active Document usingIOleObject::DoVerb(Hide, . . . ) call 1548, step IIIl. cTemplate class236 in test driver 110 informs visible plugins 150 that the currentpresentation is ending using I*::PresentationEnding( ) call 1550, stepIIIm. For example, cTemplate informs helm plugin 154 that the currentpresentation is ending using the IHelm::PresentationEnding( ) call.

Step IV, which contains sub-steps IVa-c, is the process to save pluginstate data to exam instance file 170. Test driver 110 requests the“dirty” state of plugin 150 to determine whether plugin 150 is storingany state information that would be necessary if the test were to beinterrupted. Test driver 110 uses IPersistInstance*::IsDirty( ) call1552 to make the request, step IVa. For example, test driver 110 usesIPersistInstanceSet::IsDirty call 1552 if the state data is a propertyset. If plugin 150 is storing state data that is not already stored inexam instance file 170, IPersistInstance*::IsDirty( ) call 1552 returnstrue. If plugin 150 is “dirty”, test driver 110 instructs plugin 150 tosave the state data to exam instance file 170 in the POLESS objectprovided (FIG. 26) using IPersistInstance*::Save( ) call 1554, step IVb.Finally, test driver 110 instructs plugins 150 to unload all objectsusing IPlugin::Unload( ) call 1556, step IVc.

VI. Network Environment for a Computer-Based Testing System

A description of a network environment for a computer-based testingsystem according to the present invention including a test driver thatcontrols delivery of a computer-based test over a networked environmentby caching test components for delivery to a test candidate in order tofacilitate a uniform testing environment for at least one or moreconcurrent test candidates is provided. FIG. 35 is a block diagramillustrating an example of a network environment for a computer-basedtest system according to the present invention.

With reference to FIG. 35, the computer-based testing system of thepresent invention includes a test management system, including a Webcertified proctor management site 3160 and secure ID server 3150, foridentifying proctors of the computer-based test; a test proctorcertification system, including Web eligibility management site 3190 andGEE server 3180, for certifying proctors for administering thecomputer-based test; a test registration system to enable candidates torequest an appointment to take a test, including Web registration site3200 to allow appointments to be reserved via the Internet, schedulingmiddleware 3170 to process the candidates request to take the test,schedule interface service 3220 which provides the scheduled appointmentdata to candidate workstation 3000 to identify the candidate and toverify the candidate's credentials, an ecommerce system 3210 forregistering test candidates for the computer-based test and forcollecting and/or processing any payments associated with scheduling anddelivering the computer-based test; and a test analysis system,including results collection server 3230 for aggregating results fromall delivered tests, a data warehouse 3240 for storing and allowingaccess to such test results, and a results psychometric system 3250 formonitoring candidate responses to test items, maintaining thecomposition of the computer-based tests, and to enable item responseanalysis and reports to be generated and distributed.

The network environment as shown in FIG. 35 further includes acomputer-based test production system, including test publisher system3010 for authoring a computer-based test and producing exam source files(previously described) which store test specification and content; testpackaging system 3020 for retrieving, validating and compiling the examsource files of the computer-based test into an exam resource file(previously described); and source server 3030 for storing and managingversions of the compiled exam resource file and any additionalcorresponding test components comprising the computer-based test. Thenetwork environment also includes distribution servers for facilitatingcomputer-based testing, including application deployment servers 3040,item cache servers 3050 and plugin cache servers 3060, connected tosource server 3030 via a wide area network (i.e., an interconnectedsystem of networks), and connected to candidate workstation 3000 via alocal access network. The local access network can include LAN, xDSLaccess network, cable access network, or wireless access network. Fromsource server 3030, the test components are extracted from the examresource file of the computer-based test and deployed to thedistribution servers. Application deployment servers 3040 store the testdriver application and the exam resource file, containing the core testspecifications of the computer-based test, for distribution to candidateworkstation 3000. Item cache servers 3050 store test items extractedfrom the exam resource file. Plugin cache servers 3060 store pluginsextracted from the exam resource file. The network environment furtherincludes any number of test candidate workstations at a multiplicity oflocations to which the computer-based test is scheduled to be delivered.Thus, the distribution servers may be clustered for parallel processing,load balancing and fault tolerance to support a varied volume of atleast one or more concurrent test candidates.

From application deployment servers 3040. the test driver applicationfor controlling delivery of the computer-based test may be downloadedfor setup on candidate workstation 3000. The test driver application3070 may be stored on a computer readable medium, such as a hard driveor other magnetic medium, connected to candidate workstation 3030.However, test components of the computer-based test, including the examresource file, test items and plugins (all of which were previouslydescribed), are stored in random access memory (RAM) to preventunauthorized copying or manipulation of the computer-based test, therebyassuring the integrity of the test.

Test driver application 3070 includes session manager 3080 for managingthe computer-based testing session, having authentication layer 3090 forauthenticating proctors of the computer-based test, event locatorcandidate credential interface 3100 for certifying test candidateeligibility to use the computer-based test and Unified Test Driver (UTD)core 3110 for controlling delivery of the computer-based test tocandidate workstation 3000. UTD core 3100 controls item cache controller3120, plugin cache controller 3130 and browser presentation layer 3140.Item cache controller 3120 stores test items retrieved from item cacheservers 3040 in item cache 4360 (FIG. 36), encrypted memory mapped pagedRAM on candidate workstation 3000, for delivery to the test candidate.Plugin cache controller 3130 stores plugins retrieved from plugin cacheservers 3160 in plugin cache 4390 (FIG. 36), memory mapped paged RAM oncandidate workstation 3000, for use by UTD core 3100. Browserpresentation layer 3140 serves the computer-based test on candidateworkstation 3000 in accordance to the exam resource file retrieved fromapplication deployment servers 3040.

VII. Caching Architecture for the Computer-Based Testing System

In computer-based testing, a test may be delivered to a candidateworkstation using two delivery modes: a disconnected mode (e.g.,non-networked) or a connected mode (e.g., networked). A disconnectedmode is a traditional delivery mode in which every element required toadminister the computer-based test is stored on the test candidateworkstation before the computer-based test is initiated. Thus, there isno need to monitor the testing environment to adjust delivery of thecomputer-based test over a network environment and no need to implementa caching architecture for a network environment. In contrast, aconnected mode is a non-traditional delivery mode in which core elementsof the computer-based test are stored on the test candidate workstationand additional test components are retrieved during computer-basedtesting from distribution servers over a network environment. Whileusing the connected mode provides many advantages, such as, flexibilityin administering computer-based tests to test candidates located inmultiple locations, the network environment introduces environmentvariables that are not controllable by a test administrator withoutgreat cost. Thus, in order to provide a uniform testing environment to avaried volume of at least one or more concurrent test candidates locatedin multiple locations, a computer-based system must adjust delivery of acomputer-based test in order to compensate for variance in the networkenvironment.

A description of a caching architecture for a computer-based testingsystem of the present invention including a test driver that controlsdelivery of a computer-based test over a networked environment bycaching test components for delivery to a test candidate in order tofacilitate a uniform testing environment for at least one or moreconcurrent test candidates is provided. FIG. 36 is a block diagram of acaching architecture for a computer-based test system according to thepresent invention.

As shown in FIG. 36, test driver application 3070 includes Web serviceinterface 4400, having authentication module 4410. Web service interface4400 facilitates network communications between candidate workstation3000 and the distribution servers. Authentication module 4410authenticates the test components retrieved from the distributionservers. Web service interface 4400 further logs the transmissiondetails of each request and response transmitted between candidateworkstation 3000 and the distribution servers, which includes, but isnot limited to, for example, the time at which candidate workstation3000 sends a request to a distribution server, the time at which adistribution server receives the request, the time at which adistribution server sends a response to candidate workstation 3000 andthe time at which candidate workstation 3000 receives the response. Suchdetails further include, for example, error messages issued by thenetwork, error messages issued by the distribution servers, the numberof retries by candidate workstation 3000 to send a request to thedistribution servers, the number of retries by candidate workstation3000 to obtain a response from the distribution servers and the size ofeach message or test components transmitted.

Test driver application 3070 further includes item request interface4340, plugin request interface 4370, request processor 4310, decryptionmodule 4320, decompression module 4330, item request module 4350, pluginrequest module 4380, cache controller 4400, item cache 4360 and plugincache 4390. Test driver application 3070 sends requests to retrieve testcomponents via item request interface 4340 and plugin request interface4370. Request processor 4310 processes requests for test componentsinitiated by test driver application 3070 and delivery of testcomponents retrieved from the distribution servers to test driverapplication 3070. Item request module 4360 facilitates retrieval of testitems from item cache servers 3040. Plugin request module 4380facilitates retrieval of plugins from the plugin cache servers 3060.Cache controller 4400 manages the storing of test components in itemcache 4360 and plugin cache 4390. Item cache 4350 stores test itemsretrieved from item cache servers 3040. Plugin cache 4390 stores testplugins retrieved from plugin cache servers 3060. Decryption module 4320decrypts test components that have been encrypted for preserving theintegrity of the computer-based test. Decompression module 4330decompresses test components that have been compressed for transmittingover the network environment.

Descriptions of example caching operations of a computer-based testingsystem of the present invention including a test driver that controlsdelivery of a computer-based test over a networked environment havingthe caching architecture are provided. Because the computer-based testcomprises cacheable objects, it is possible to download only selectedtest components from the distribution servers to the candidateworkstation for delivering a current test section in accordance to thetest specifications. Thus, it is possible for a test candidate toinitiate a computer-based test prior to all test components beingdownloaded to the candidate workstation.

FIG. 37A is a flow chart of a method of caching test items with respectto the caching architecture according to the present invention. Withreference to FIG. 37A, when test driver application 3070 requires a testitem, test driver application 3070 via item request interface 4340instructs request processor 4310 to retrieve the test item at stepsS3710A and S3715A. Request processor 4310 verifies whether the test itemis available in item cache 4360 at step S3720A. If the requested testitem is available locally (e.g., hit), request processor 4310 retrievesthe test item from item cache 4360 at step S3755A. If necessary,decompression module 4330 decompresses the test item at step S3750A anddecryption module 4320 decrypts the test item at step S3765A. Requestprocessor 4310 returns the test item to test driver application 3070 atstep S3770A. However, if the test item is not available locally (e.g.,miss), item request module 4350 requests the test item from item cacheservers 3050 via Web service interface 4400 at steps S3725A and S3730A.Web service interface 4400 requests the test item from item cache server3050 at step S3730A. Item cache server 3050 or the source server 3030returns the requested test item at step S3735A. Authentication module4410 authenticates the returned requested test item at step S3740A. Itemcache controller 3120 requests storage instructions from cachecontroller 4400 at step 3745A and accordingly stores the returnedrequested test item in item cache 4360 at a step S3750A. Requestprocessor 4310 retrieves the test item from item cache 4360 at stepS3755A. If necessary, decompression module 4330 decompresses the testitem at step S3760A and decryption module 4320 decrypts the test item atstep S3765A. Request processor 4310 delivers the test item to testdriver application 3070 at step S3770A.

A similar process occurs when test driver application 3070 requires aplugin. FIG. 37B is a flow chart of a method of caching plugins respectto the caching architecture according to the present invention. Withreference to FIG. 37B, when test driver application 3070 requires aplugin, test driver application 3070 via plugin request interface 4370instructs request processor 4310 to retrieve the plugin at steps S3710Band S3715B. Request processor 4310 verifies whether the plugin isavailable in plugin cache 4390 at step S3720B. If the requested pluginis available locally (e.g., hit), request processor 4310 retrieves theplugin from plugin cache 4390 at step S3755B. If necessary,decompression module 4330 decompresses the plugin at step S3760B anddecryption module 4320 decrypts the plugin at step S3765B. Requestprocessor 4310 returns the plugin to test driver application 3070 atstep S3770B. However, if the plugin is not available locally (e.g.,miss), plugin request module 4370 requests the plugin from plugin cacheservers 3060 via Web service interface 4400 at steps S3725B and S3730B.Web service interface 4400 requests the plugin from plugin cache server3060 at step S3730B. Plugin cache server 3060 or the source server 3030returns the requested plugin at step S3735B. Authentication module 4410authenticates the returned requested plugin at step S3740B. Plugin cachecontroller 3130 requests storage instructions from cache controller 4400at step S3745A and accordingly stores the returned requested plugin inplugin cache 4390 at a step S3750B. Request processor 4310 retrieves theplugin from plugin cache 4390 at step S3755B. If necessary,decompression module 4330 decompresses the plugin at step S3760B anddecryption module 4320 decrypts the plugin at step S3765B. Requestprocessor 4310 delivers the plugin to test driver application 3070 atstep S3770B. Moreover, a cache cleaning algorithm may be employed toretire test items and/or plugins when one or more predeterminedconditions are satisfied, e.g., when a predetermined amount of timeexpires after the test item and/or the plugin is added to item cache4360 or plugin cache 4390, respectively.

Caching of a computer-based test for delivery to candidate workstation3000 is facilitated in accordance to the changing testing environmentduring computer-based testing reflected by monitoring of candidateprogress, candidate performance, network bandwidth, network latency andserver response, among other environmental variables, duringcomputer-based testing. With reference to FIG. 36, test driverapplication 3070 further includes stimuli processor 4400, cachecontroller 4410, candidate performance monitor 4420, candidate progressmonitor 4430, network latency monitor 4440, network bandwidth monitor4450 and server response monitor 4460. In response to changes in thetesting environment, stimuli processor 4410 adjusts the source forretrieving test components or the volume of test components to store incache memory on candidate workstation 3000 and instructs cachecontroller 4410 accordingly. Environment monitoring means as shown inFIG. 36 include, but are not limited to, for example, candidate progressmonitor 4430, which measures the rate at which the test candidate isanswering test items; candidate performance monitor 4420, which measurestest candidate competency; network bandwidth monitor 4450, whichmeasures data transfer speed between candidate workstation 3000 anddistributions servers; network latency monitor 4440, which measuresdelay times between candidate workstation 3000 and distributions serverscaused by the network; and server response monitor 4460, which measuresthe delay times between candidate workstation 3000 and distributionsservers when delay is caused by the server.

Descriptions of caching operations of a computer-based testing systemaccording to the present invention is provided. Generally, stimuliprocessor 4410 periodically initiates an inquiry to candidate progressmonitor 4430, candidate performance monitor 4420, network bandwidthmonitor 4450, network latency monitor 4440 and server response monitor4460 during computer-based testing. The results of each monitor arereturned to stimuli processor 4410. Based these results, stimuliprocessor 4410 adjusts either the source of test components or thevolume of test components being cached for delivery of thecomputer-based test and cache controller 4400 accordingly. Examples ofthe operations of the testing environment monitors candidate progressmonitor 4430, candidate performance monitor 4420, network bandwidthmonitor 4450, network latency monitor 4440 and server response monitor4460 are now described.

For example, candidate progress monitor 4430 measures the testcandidate's rate of progress in answering test items duringcomputer-based testing for maintaining availability of test items. FIG.39A shows a flow chart of the operation of candidate progress monitor4430. Stimuli processor 4410 initiates an inquiry to candidate progressmonitor 4430 at step S3800A. Candidate progress monitor 4430 retrievesthe number of test items allotted to the current test section and thetime allotted to each test item, which are stored in the exam resourcefile (described previously) on candidate workstation 3000 at stepS3810A. Candidate progress monitor 4430 retrieves the number of testitems answered by the test candidate and the answer time used by thetest candidate for answering each test item, which are stored in theexam instance file (described previously) on candidate workstation 3000at step S3815A. Candidate progress monitor 4430 then calculates the rateat which the test candidate answers test items at step S3820A andreturns the number of test items allotted to the current test section,the number of test items answered by the test candidate in the currenttest section and the answer rate to stimuli processor 4410 at stepS3825A for determining whether a sufficient pool of test items areavailable to the test candidate during computer-based testing. Based onthese results, stimuli processor 4410 calculates the number of testitems remaining in the current test section for delivery to the testcandidate at S3830A. Stimuli processor 4410 then retrieves the number oftest items stored in item cache 4360 at step S3835A and calculates thenumber of test items remaining to be cached in the current test sectionfor delivery to the test candidate at step S3840A. Based on the answerrate and the number of test items remaining to be cached, stimuliprocessor 4410 determines whether it has become necessary to retrieveadditional test items for caching at step S3845A. When it has becomenecessary to retrieve additional test items, stimuli processor 4410instructs cache controller 4400 accordingly at step S3850A. Otherwise,monitoring continues at step S3855A.

In another example, candidate performance monitor 4420 measures the testcandidate's competency in answering test items during computer-basedtesting for maintaining availability of suitable test items. FIG. 39Bshows a flow chart of the operation of candidate performance monitor4420. Stimuli processor 4410 initiates an inquiry to candidateperformance monitor 4420 at step S3800B. Candidate performance monitor4420 retrieves the number of test items answered in the current testsection, the competency level of each test item and the scoring of eachtest item, which are stored in the exam instance file (describedpreviously) on the candidate workstation at step S3810B. Candidateperformance monitor 4420 then calculates the competency level of thetest candidate at step S3815B and returns the candidate competency levelto stimuli processor 4410 at step S3820B for determining whether asufficient pool of suitable test items are available to the testcandidate during computer-based testing. Stimuli processor 4410 thenretrieves the number of test items of the candidate competency levelstored in item cache 4360 at step S3825B and calculates the number oftest items of the candidate competency level remaining to be cached inthe current test section for delivery to the test candidate at S3830B.Based on the candidate competency level and the number of test items ofthe candidate competency level remaining to be cached, stimuli processor4410 determines whether it has become necessary to retrieve additionaltest items of the current competency level for caching at step S3835B.When it has become necessary to retrieve additional test items of thecandidate competency level, stimuli processor 4410 instructs cachecontroller 4400 accordingly at step S3840B. Otherwise, monitoringcontinues at step S3845B.

A further example is network bandwidth monitor 4450 which measures thespeed of data transfer between candidate workstation 3000 and adistribution server for maintaining timely availability of test items.FIG. 39C shows a flow chart of the operation of network bandwidthmonitor 4450. Stimuli processor 4410 initiates an inquiry to networkbandwidth monitor 4450 at step S3800C. For each exchange betweencandidate workstation 3000 and a distribution server, network bandwidthmonitor 4450 retrieves from logs generated by Web service interface 4400the time a request is sent by candidate workstation 3000 to adistribution server, the time the request is received by thedistribution server, the size of the request message, the time aresponse is sent by the answering distribution server, the time theresponse is received by candidate workstation 3000 and the size of theresponse message at step S3810C. Based on these variables, networkbandwidth monitor 4450 calculates the data transfer speeds betweencandidate workstation 3000 and distribution servers at step S3015C andreturns the data transfer speeds to stimuli processor 4410 at stepS3820C for determining frequency of requests to retrieve test componentsand size of test components being transferred. Based on these results,stimuli processor 4410 calculates whether network bandwidth is growingor deteriorating at step S3825C and then stimuli processor 4410instructs cache controller 4400 accordingly at step S3830C. For example,when network bandwidth is narrowing, frequency of data transfer may beincreased and size of test components to be transferred may be decreasedto compensate for loss of data transfer speed at step S3835C. Otherwise,monitoring continues at step S3840C.

An additional example, network latency monitor 4440 measures the delayin data transmission time between candidate workstation 3000 and thedistribution servers caused by the network for re-establishingconnectivity during computer-based testing. FIG. 39D shows a flow chartof the operation of network latency monitor 4440. Stimuli processor 4410initiates an inquiry to network latency monitor 4440 at step S3800D. Foreach exchange between candidate workstation 3000 and a distributionserver, network latency monitor 4440 retrieves from logs generated byWeb service interface 4400 the time a request is sent by candidateworkstation 3000 to a distribution server, the time the request isreceived by the distribution server, the size of the request message,the time a response is sent by the answering distribution server, thetime the response is received by candidate workstation 3000, the size ofthe response message, the number of transmission retries and any networkerror messages received at step S3810D. Based on these variables,network latency monitor 4440 calculates delay times between candidateworkstation 3000 and distribution servers caused by the network at stepS3815D and returns the delay times to stimuli processor 4410 at stepS3820D for determining the state of network connectivity. Stimuliprocessor 4410 retrieves a predetermined tolerance threshold from theexam resource file (described previously) at step S3825D and determinesthe rate delay times are approaching the tolerance threshold at stepS3830D. Delay times approaching within a predetermined range of thetolerance threshold indicates that network connectivity failure isimminent. Prior to such failure, stimuli processor 4410 determineswhether a sufficient number of test components are cached for deliveryto the test candidate while the test driver reconnects to a redundantnetwork in the background at step S3835D and instructs cache controller4400 accordingly at step S3840D. Otherwise, monitoring continues at stepS3845D.

In another example, server response monitor 4460 measures the delay indata transmission time between candidate workstation 3000 and thedistribution servers caused by the distribution servers for maintainingaccessibility to a source for test components during computer-basedtesting. FIG. 39E shows a flow diagram of the operation of serverresponse monitor 4460. Stimuli processor 4410 initiates an inquiry toserver response monitor 4460 at step S3800E. For each exchange betweencandidate workstation 3000 and a distribution server, server responsemonitor 4460 retrieves from logs generated by Web service interface 4400the time a request from candidate workstation 3000 is received by adistribution server, time a response is sent by the answeringdistribution server, the number of retries and any server error messagesreceived at step S3810E. Based on these variables, server responsemonitor 4460 calculates delay times between candidate workstation 3000and a distribution server caused by the servers at step S3815E andreturns the delay times to stimuli processor 4410 at step S3820E fordetermining the state of server accessibility. Stimuli processor 4410retrieves a predetermined tolerance threshold from the exam resourcefile (described previously) at step S3825E and determines the rate delaytimes are approaching the tolerance threshold at step S3830E. Delaytimes approaching within a predetermined range of the tolerancethreshold indicates that server accessibility is limited and stimuliprocessor 4410 instructs cache controller 4400 accordingly at stepS3835E. Otherwise, monitoring continues at step S3840E.

The present invention is not limited to the embodiment described herein.For example, the test driver application and cacheable test componentsmay be stored on the same distribution servers for delivery to testcandidates. In addition, the distribution servers, in groups or singly,can be located in any number of remote locations. Furthermore, thetesting environment monitors can include other monitors not specificallydescribed herein. Thus, cacheable objects are used to expandfunctionality of a test driver application that controls delivery of acomputer-based test to one or more test candidates over a dynamicdistributed network environment by adapting delivery of thecomputer-based test in accordance to monitoring of testing environmentvariables.

The many features and advantages of the invention are apparent from thedetailed specification, and thus, it is intended by the appended claimsto cover all such features and advantages of the invention, which fallwithin the true spirit and scope of the invention. Further, sincenumerous modifications and variations will readily occur to thoseskilled in the art, it is not desired to limit the invention to theexact construction illustrated and described, and accordingly, allsuitable modifications and equivalence may be resorted to, fallingwithin the scope of the invention.

1. A system for computer-based testing comprising: (a) delivery storagemeans for storing at least one computer-based test and delivering overan electronic data network said computer-based test to at least one testcandidate; (b) test driver means for controlling delivery of saidcomputer-based test from said delivery storage means; and (c) cachestorage means for caching said computer-based test in response tomonitoring of at least one testing environment variable.
 2. Thecomputer-based testing system of claim 1, wherein said delivery storagemeans comprises a plurality of test distribution servers.
 3. Thecomputer-based testing system of claim 2, wherein said plurality of testdistribution servers comprises: (a) program deployment means for storingsaid test driver means for delivery to said test candidate; (b) dataobject deployment means for storing cacheable data objects of saidcomputer-based test for delivery to said test candidate; and (c)application object deployment means for storing cacheable applicationobjects of said computer-based test for delivery to said test candidate.4. The computer-based testing system of claim 3, wherein said cacheabledata objects comprise text, multimedia and template objects.
 5. Thecomputer-based testing system of claim 3, wherein said cacheableapplication objects comprise plugin program objects.
 6. Thecomputer-based testing system of claim 1, wherein said cache storagemeans comprises first and second storage means.
 7. The computer-basedtesting system of claim 6, wherein said first storage means comprisesmagnetic storage medium.
 8. The computer-based testing system of claim6, wherein said second storage means comprises random access memory(RAM).
 9. The computer-based testing system of claim 1, wherein saidtest driver means comprises: (a) a session management component foradministering said computer-based test to said test candidate; (b) aproctor authentication component for identifying at least one proctor ofsaid computer-based test; (c) a scheduling component for verifying testcandidate registration for said computer-based test; (d) a test driverprogram for controlling delivery of said computer-based test; (e) afirst cache controller for controlling storing of cacheable data objectsin encrypted RAM; (f) a second cache controller for controlling storingof cacheable application objects in RAM; and (g) a browser presentationcomponent for serving said computer-based test to said test candidate.10. The computer-based testing system of claim 9, wherein said testdriver program comprises: (a) a first request interface for requestingcacheable data objects; (b) a second request interface for requestingcacheable application objects; (c) a request processor for processingrequests of said test driver program to retrieve cacheable data objectsand cacheable application objects; (d) a decryption module fordecrypting cacheable data objects and cacheable application objects; (e)a decompression module for decompressing cacheable data objects andcacheable application objects; (f) a first request module for retrievingcacheable data objects; (g) a second request module for retrievingcacheable application objects; (h) a first cache for storing retrievedcacheable data objects; (i) a second cache for storing retrievedcacheable application objects; (j) a stimuli processor for adaptingdelivery of said computer-based test; (k) a cache controller forcontrolling source of retrieving cacheable data objects and cacheableapplication objects and for controlling volume of cacheable data objectsand cacheable application objects stored in cache storage means; (l) anetwork interface for communicating with a plurality of testdistribution servers; and (m) an authentication component forauthenticating cacheable data objects and cacheable application objects.11. The computer-based testing system of claim 1, wherein said testdriver means comprises at least one monitoring means for monitoring atesting environment variable.
 12. The computer-based testing system ofclaim 11, wherein said monitoring means is a test candidate progressmonitor for measuring progress of said test candidate duringcomputer-based testing.
 13. The computer-based testing system of claim11, wherein said monitoring means is a test candidate performancemonitor for measuring competency of said test candidate duringcomputer-based testing.
 14. The computer-based testing system of claim11, wherein said monitoring means is a network bandwidth monitor fordetermining a speed of a network connection through which saidcomputer-based test is being delivered.
 15. The computer-based testingsystem of claim 11, wherein said monitoring means is a network statemonitor for determining a state of a network connection through whichsaid computer-based test is being delivered.
 16. The computer-basedtesting system of claim 11, wherein said monitoring means is a serverstate monitor for determining a state of a server from which saidcomputer-based test is being delivered.
 17. The computer-based testingsystem of claim 1, wherein said electronic data network is the Internet.18. A method for computer-based testing comprising the steps of: (a)storing at least one computer-based test and delivering over anelectronic data network said computer-based test to at least one testcandidate by delivery storage means; (b) controlling delivery of saidcomputer-based test from said delivery storage means by test drivermeans; and (c) caching and storing said computer-based test in cachestorage means in response to monitoring of at least one testingenvironment variable.
 19. The method for computer-based testing of claim18, the method comprising the step of storing on a plurality of testdistribution servers said computer-based test for delivery to aplurality of test candidates.
 20. The method for computer-based testingof claim 19, the method comprising the step of storing cacheable dataobjects comprising text, multimedia and template objects.
 21. The methodfor computer-based testing of claim 19, the method comprising the stepof storing cacheable application objects comprising plugin programobjects.
 22. The method for computer-based testing of claim 19, themethod comprising the step of storing said test driver means in firststorage means comprising magnetic storage medium.
 23. The method forcomputer-based testing of claim 18, the method comprising the step ofstoring said cacheable data objects in second storage means comprisingrandom access memory (RAM).
 24. The method for computer-based testing ofclaim 18, the method comprising the step of storing said cacheableapplication objects in second storage means comprising random accessmemory (RAM).
 25. The method for computer-based testing system of claim18, the method comprising the steps of: (a) administering saidcomputer-based test to said test candidate using a session managementcomponent; (b) identifying at least one proctor of said computer-basedtest using a proctor authentication component; (c) verifying testcandidate registration for said computer-based test using a schedulingcomponent; (d) controlling delivery of said computer-based test using atest driver program; (e) controlling storing of cacheable data objectsin encrypted RAM using a first cache controller; (f) controlling storingof cacheable application objects in RAM using a second cache controller;and (g) serving said computer-based test to said test candidate using abrowser presentation component.
 26. The method of computer-based testingof claim 25, the method comprising the steps of: (a) requestingcacheable data objects using a first request interface; (b) requestingcacheable application objects a second request interface; (c) processingrequests of said test driver program to retrieve cacheable data objectsand cacheable application objects using a request processor; (d)decrypting cacheable data objects and cacheable application objectsusing a decryption module; (e) decompressing cacheable data objects andcacheable application objects using a decompression module; (f)retrieving cacheable data objects using a first request module; (g)retrieving cacheable application objects using a second request module;(h) storing retrieved cacheable data objects using a first cache; (i)storing retrieved cacheable application objects using a second cache;(j) adapting delivery of said computer-based test using a stimuliprocessor; (k) controlling source for retrieval of cacheable dataobjects and cacheable application objects and for controlling volume ofcacheable data objects and cacheable application objects stored in cachestorage means using a cache controller; (l) communicating with aplurality of test distribution servers using a network interface; and(m) authenticating cacheable data objects and cacheable applicationobjects using an authentication component.
 27. The method ofcomputer-based testing of claim 18, the method comprising the step ofmonitoring at least one test environment variable.
 28. The method ofcomputer-based testing of claim 27, the method comprising the step ofmonitoring test candidate progress for measuring progress of said testcandidate during computer-based testing.
 29. The method ofcomputer-based testing of claim 27, the method comprising the step ofmonitoring test candidate performance for measuring competency of saidtest candidate during computer-based testing.
 30. The method ofcomputer-based testing of claim 27, the method comprising the step ofmonitoring network bandwidth for determining a speed of a networkconnection through which said computer-based test is being delivered.31. The method of computer-based testing of claim 27, the methodcomprising the step of monitoring network state for determining a stateof a network connection through which said computer-based test is beingdelivered.
 32. The method of computer-based testing of claim 27, themethod comprising the step of monitoring server state for determining astate of a server from which said computer-based test is beingdelivered.
 33. The method for computer-based testing of claim 18, themethod comprising the step of delivering over the Internet saidcomputer-based test.
 34. A computer-readable storage medium for storinga computer-based testing method, said computer-based testing methodcomprising: (a) storing at least one computer-based test on deliverystorage means and delivering over an electronic data network saidcomputer-based test to at least one test candidate by said deliverystorage means; (b) controlling delivery of said computer-based test fromsaid delivery storage means by test driver means; and (c) caching andstoring said computer-based test in cache storage means in response tomonitoring of at least one testing environment variable.
 35. The storagemedium for storing a computer-based testing method of claim 34, themethod comprising the step of storing on a plurality of testdistribution servers said computer-based test for delivery to aplurality of test candidates.
 36. The storage medium for storing acomputer-based testing method of claim 35, the method comprising thestep of storing cacheable data objects comprising text, multimedia andtemplate objects.
 37. The storage medium for storing a computer-basedtesting method of claim 35, the method comprising the step of storingcacheable application objects comprising plugin program objects.
 38. Thestorage medium for storing a computer-based testing method of claim 35,the method comprising the step of storing said test driver means infirst storage means comprising magnetic storage medium.
 39. The storagemedium for storing a computer-based testing method of claim 35, themethod comprising the step of storing said cacheable data objects insecond storage means comprising random access memory (RAM).
 40. Thestorage medium for storing a computer-based testing method of claim 35,the method comprising the step of storing said cacheable applicationobjects in second storage means comprising random access memory (RAM).41. The storage medium for storing a computer-based testing method ofclaim 34, the method comprising the steps of: (a) administering saidcomputer-based test to said test candidate using a session managementcomponent; (b) identifying at least one proctor of said computer-basedtest using a proctor authentication component; (c) verifying testcandidate registration for said computer-based test using a schedulingcomponent; (d) controlling delivery of said computer-based test using atest driver program; (e) controlling storing of cacheable data objectsin encrypted RAM using a first cache controller; (f) controlling storingof cacheable application objects in RAM using a second cache controller;and (g) serving said computer-based test to said test candidate using abrowser presentation component.
 42. The storage medium for storing acomputer-based testing method of claim 41, the method comprising thesteps of: (a) requesting cacheable data objects using a first requestinterface; (b) requesting cacheable application objects a second requestinterface; (c) processing requests of said test driver program toretrieve cacheable data objects and cacheable application objects usinga request processor; (d) decrypting cacheable data objects and cacheableapplication objects using a decryption module; (e) decompressingcacheable data objects and cacheable application objects using adecompression module; (f) retrieving cacheable data objects using afirst request module; (g) retrieving cacheable application objects usinga second request module; (h) storing cacheable data objects using afirst cache; (i) storing cacheable application objects using a secondcache; (j) adapting delivery of said computer-based test using a stimuliprocessor; (k) controlling source for retrieval of cacheable dataobjects and cacheable application objects and for controlling volume ofcacheable data objects and cacheable application objects stored in cachestorage means using a cache controller; (l) communicating with aplurality of test distribution servers using a network interface; and(m) authenticating cacheable data objects and cacheable applicationobjects using an authentication component.
 43. The storage medium forstoring a computer-based testing method of claim 34, the methodcomprising the step of monitoring at least one test environmentvariable.
 44. The storage medium for storing a computer-based testingmethod of claim 43, the method comprising the step of monitoring testcandidate progress for measuring progress of said test candidate duringcomputer-based testing.
 45. The storage medium for storing acomputer-based testing method of claim 43, the method comprising thestep of monitoring test candidate performance for measuring competencyof said test candidate during computer-based testing.
 46. The storagemedium for storing a computer-based testing method of claim 43, themethod comprising the step of monitoring network bandwidth fordetermining a speed of a network connection through which saidcomputer-based test is being delivered.
 47. The storage medium forstoring a computer-based testing method of claim 43, the methodcomprising the step of monitoring network state for determining a stateof a network connection through which said computer-based test is beingdelivered.
 48. The storage medium for storing a computer-based testingmethod of claim 43, the method comprising the step of monitoring serverstate for determining a state of a server from which said computer-basedtest is being delivered.
 49. The storage medium for storing acomputer-based testing method of claim 43, the method comprising thestep of delivering over the Internet said computer-based test.